It depends on the context, but generally "in doing so" refers to something that happens along with or as part of the action, and "by doing so" refers to a result of the action. In many cases either one works. Here are a couple of cases where one is preferable to the other:
"Japan bombed Pearl Harbor and in doing so sank several American ships." It's part of the same action, so you wouldn't use "by doing do."
"Japan bombed Pearl Harbor and by doing so made bombing attacks on Japan inevitable." It's a later consequence, so you wouldn't use "in doing so." This is less clear than the first example, though.
"Japan bombed Pearl Harbor and in/by doing so entered war with the US." In this case you can view starting war either as an aspect or a consequence of the attack, so I think either one is appropriate.