Repo: Tips & Tricks

Repo is one of a triplet of tools that you will use when you work with Android source, the other two being Git and Gerrit. You can read more about those tools from Android official site.

This article focuses on repo and aims at  teaching you some useful tips & tricks that will hopefully help you become more efficient when managing Android source.

What is Repo?

Repo is a repository management tool built on top of Git.

It’s first purpose is to downloads files from multiple git repositories into your local working directory (aka the source tree). That means that you won’t have to manually download or fetch the latest changes from about 300+ projects (depending on if you are downloading AOSP, CM or AOKP for example), repo  will do it for you.

The second purpose of repo is to made it easy to submit your code contributions for review to a Gerrit server.

Downloading Android Source: 3 easy steps

The first thing you will always read about Android source is how to download them to your local hard drive. It’s obviously the first stage of working with the sources. Fetching the sources is as easy as the three following steps:

  • The first step is to download the repo command and make it executable:

Check your shell login scripts (like .profile, .bash_profile,…) that ~/bin is included in your $PATH.

  • The second step will initialiaze a new local source repository in the current working directory:

This will create a new place to hold your local copy of the source tree. “url” should point to a Manifest repository that describes the whole sources. It is a special project with a file (default.xml) that lists all the projects that Android is made of. In the Manifest file, each projects has attributes about: where to place it in the tree, where to download it from (git server), revision that will be used (usually a branch name, tag or commit sha-id). An example of such file can be found here.

  • The third and last step is to actually download the source:

The first time you run this command, it will download the sources. Android sources are huge, for example, CM10 (Jellybean) sources are about 14GiB. So, be prepared: you’ll need space in your hard drive, and time (possibly several hours).

Hopefully this article will help you save both time and space with some useful tricks. Next times you run it, it will only fetch the latest changes made to each repository.

Looking inside Repo

Let’s get back to the second step we’ve just seen earlier. What does repo init  really do? repo init  will create a hidden directory called  .repo/  in the root of your source tree. Inside .repo/  we will find very interesting things:

  • repo/ : That is actually a (git) clone of the full repo  sources themselves, while the repo command we downloaded earlier is much more a launcher that will then rely on those. It’s made in Python.
  • manifest.xml : The main manifest of the tree (usually android/default.xml). DO NOT edit this file.
  • projects/ : This folder will actually hold the .git structures of every projects you repo sync. We will see later how it will save us several hours.
  • local_manifest.xml : That file won’t be created by repo, but it’s relevant to know it. It’s an optional file you can create to override the main manifest.

Downloading the source efficiently

Considering the size of Android sources, more than 250.000 files, with all the versions of those files, we are talking of about 14GiB for CM10. It can take a lot of time to download! Fortunately, you can instruct repo to download several projects in paralell using the “-j” flag, so therepo sync  command will read something like:

You should tune the number of jobs repo will launch in paralell with regards of your number of CPU cores, and your bandwidth. -j 16 should be fine for a 8-cores box with a high Internet bandwidth.

This option will considerably lower the time needed to download the sources. Althought, be careful, as raising this number too much might be counter productive.

To avoid issues, it is recommended that you also add  net.ipv4.tcp_window_scaling=0   to your /etc/sysctl.conf   file (and run sudo sysctl -p  , or reboot to apply the change).

Another handy option you might consider using is the “-c” flag:

We’ve seen that Android source is made of hundreds of project, each project having it’s sources in its own git repository. With all the Android versions jumps, those projects have evolved, and all that history is still recorded in those git repositories. That means that each of those hundred git repository will have several branches. So the question is: do we really need to download (git-clone) all the branches of every projects that make Android? Probably not.

That is where the “-c” flag is useful. With that flag, repo will only download the revision (branch) that is specified in the manifest, not all the branches that are on the remote server. It will thus save us quite some space, and again it will take less time to download.

Editing the default Manifest

We’ve already seen that repo relies on a manifest file, usually a file called default.xml in the android project. The copy that repo  will use as its reference is saved as .repo/manifest.xml.

We’ve also said that we shouldn’t modify the .repo/manifest.xml manually!

The right way to modify the manifest.xml file would be to edit the android/default.xml, commit the change, and push it upstream and thenrepo sync . Doing so will update the .repo/manifest.xml copy, and everything will stay under (git & repo) control.

Overriding the default Manifest: Local Manifest

There can be cases where editing the android/default.xml file isn’t the best option, or not an option at all. For example, if we have downloaded the sources from a Team, and we don’t have access to pushing to their git. Or because the change is relevant only to me, or only to a device, or maybe it’s just for testing purpose.

Think also about the two following  questions:

  • Do you really need all the projects from the default.xml?
  • Do you need any other projects not in the default.xml?

For example, when building on a Linux machine, do you really need to download the toolchains for building arm binaries on a darwin machine?! I guess not.

In those cases, we can use the “local manifest” to override (modify) the default manifest. It’s a file calledlocal_manifest.xml , under .repo/ in our source tree. The file doens’t exist, we will need to create it.

This local_manifest.xml  will allow us to:

  • Add new “remotes”
  • Add, remove or replace “projects” (repositories)

Obviously, this file won’t be modified with repo sync., that is the point of it.

The synthax is the same as for the default manifest:

  • <remote/>    to add a remote
  • <project />    to add a project
  • <remove-project />   to remove a project

Replacing a project with another is simply done by removing the project from the default manifest and add a new one in its place, like for example:

See repo help manifest  for more information.

Save a snapshot of the source tree

As we’ve said earlier, the source tree is described in a manifest file that is a combination of the android/default.xml and an eventual local_manifest.xml file you might have created.

repo  allows us to export the resulting manifest. It will describes the current state of our source tree (any project and their revision at the current time). The command to achive this is:

You can of course save the snapshot.xml wherever you want, and change the name to your liking.

It can be very hand to save a snapshot of our source tree each time you make a new build, so you can always come back to that exact same point later on if needed.

To restore the tree to the state described in a manifest snapshot file we had taken previously, we would run the following commands:

It’s however important to take into account some considerations here:

  • Any modification you might have made to any project which are not yet commited won’t be “saved” in the manifest snapshot file.
  • If you have commited some changes locally, but haven’t pushed those to the upstream remote repository, those  commits will be saved in the manifest snapshot, but you will only be able to restore the manifest snapshot locally.

Removing a project

Sometimes you might want to completely remove a project from your source tree. The first way that may come into your mind would be to simply remove the directory you see in your source tree.

Remember that when repo syncs the source tree, it will first clone all the git repositories under .repo/projects, then fetch the files of every one of those projects into the source tree directory. So, if you want to completely remove a project from your local sources, you will also need to find the git directory corresponding to that project under .repo/projects and delete it.

Lets see an example: we want to completely remove hardware/qcom/display . The corresponding git directory is .repo/projects/hardware/qcom/display.git :

Syncing a full new source tree in minutes!

Time might come when you’ll need to work with several full source trees. For example, you may want to have an AOKP source tree as well as a CM tree.

Obvisouly, you could repeat the same steps, to repo init and then repo sync -j 8 -c for both. But that would be twice as time consuming.

There is an alternative that should be much faster. Lets say that you have already initialized and synced a CM source tree, in ~/android/cm.

Now, we will prepare a new repo for AOKP tree, in ~/android/aokp. Here comes the trick: We will use the  --reference  flag of repo init  to indicate to repo that we already have a local repository that it can take information from, instead of downloading everything from Internet again:

You should notice a big time diference, from hours for a full repo sync, down to minutes using this trick!

Troubleshooting Repo

Like happen with all piece of software, repo can hang, or appear not to work like you would expect…

  • Kill a hanging repo command

If a repo sync  have been stuck for too long, you can kill it easily issuing the following command:

Obvisouly, you should only run that command if you know there won’t be any other python script running on the machine at that time (with the same user).

  • Make repo be more verbose

Sometimes you’ll wish repo be more verbose. It’s particularly true when repo sync fail, but won’t tell you why! The reason is usually a remote git server that isn’t available. To make repo be more verbose about what it does, you can add the --trace flasg, right after repo:

  • Get help

You can always read repo’s help, for any command, for example: