We have encountered an issue with submodule.update that reproduces in a rather special circumstance:

Setup
Have a git repo, say test, containing one submodule, say test_submodule. test_submodule must have no master branch (let say the default branch is called develop instead).

Check out a revision of test that points to the tip of develop for test_submodule, but do not git submodule init the submodule. The directory hierarchy should look like this:

test/
   ... contents of repo ...
  test_submodule/
     <empty>

Now, create a Repo for test, and call repo.submodules[0].update(init=True)

Expected results
The tip of develop is checked out in test_submodule and the index is clean.

Actual results
test_submodule is pointing at the tip of develop, but all of the files are staged for deletion. Additionally, there is a warning printed:

Failed to checkout tracking branch refs/heads/master

My analysis of why this happens
Before any bad behavior happens, submodule.update clones the submodule repository with -n, which means the clone does not actually check out the commit the clone ends up with. Remember this for later.

After cloning, we begin the process of updating the submodule to match what the parent repo specifies. submodule.update takes an optional branch argument, which defaults to None. When branch is None, we assume the branch is master. Here, we try to find this branch and point HEAD to it, but of course in the repro case this fails because the branch does not exist. This is crucial, because this means we skip the line that marks the repo as "not checked out" by pointing the branch to the "NULL" sha.

Now, recall that a requirement of the repro is that test points to the tip of develop for test_submodule. Since we did not move the repo to the NULL sha beforehand, the repo is already at the desired sha when we arrive at this conditional. Therefore, the condition evaluates to false, and we skip all the code that actually checks out code.

Finally, recall our -n checkout from the first paragraph. Since we did not check out any code after cloning, the repo is left in an un-checked-out state, which is exactly the "Actual results" state described above.

Closing thoughts
We can workaround this issue simply by adding a dummy master branch to test_submodule, but this should not be required. Ideally, submodule.update does not require a valid branch to operate correctly, since git submodule update --init works just fine without one.