It depends on the context, but generally "in doing so" refers to something that happens along with or as part of the action, and "by doing so" refers to a result of the action. In many cases either one works. Here are a couple of cases where one is preferable to the other:
- Japan bombed Pearl Harbor and in doing so sank several American ships."
It's part of the same action, so you wouldn't use "by doing so."
- Japan bombed Pearl Harbor and by doing so made bombing attacks on Japan inevitable.
It's a later consequence, so you wouldn't use "in doing so." This is less clear than the first example, though.
- Japan bombed Pearl Harbor and in/by doing so entered war with the US.
In this case you can view starting war either as an aspect or a consequence of the attack, so I think either one is appropriate.