After thinking about the backend side of challenge handling for a long time, I eventually realized the whole model was basically Git plus CI/CD. Challenge authors push challenge definitions through Git, the platform automatically runs the build step according to the checker type, and the build result determines whether the challenge is ready. Then, when players access the challenge, the platform can serve it directly. With that model, the entire challenge service becomes highly automated. Challenge authors only need to write a build script, fill in the challenge-specific config files, and push.
But there was one problem: Rust does not really have a crate that can directly act as a remote Git service. There is a binding for libgit2, but libgit2 itself does not provide server-side support. There is also gixoide, but most of it was still alpha at the time…
So I hand-rolled an HTTP implementation based on the Git documentation. Then the next big question showed up: what about the internal Git protocol? I was not about to reimplement Git from scratch… When in doubt, look at how existing projects do it. So I opened Gitea. Gitea’s answer was: you can just subprocess.popen("git")…
Transport protocol
The first step was implementing fetch and push so challenge authors could interact with repositories on the competition platform using plain Git. According to the Git transport protocol internals, a remote interaction starts with a negotiation over reference data. Taking git-fetch as an example, the client begins by sending an HTTP GET request to the server:
=> GET $GIT_URL/info/refs?service=git-upload-pack
001e# service=git-upload-pack
00e7ca82a6dff817ec66f44342007202690a93763949 HEAD multi_ack thin-pack \
side-band side-band-64k ofs-delta shallow no-progress include-tag \
multi_ack_detailed no-done symref=HEAD:refs/heads/master \
agent=git/2:2.1.1+github-607-gfba4028
003fca82a6dff817ec66f44342007202690a93763949 refs/heads/master
0000During this first exchange, the client asks the server for the reference list. The server queries the repository through a git-upload-pack process, organizes the objects it has into a list, and sends that list back to the client. The very first line also appends the set of capabilities supported by the server.
Once the client has the server’s object list, it inspects its own local repository, computes the difference against the server, and combines that information into a second request. It then sends an HTTP POST request:
=> POST $GIT_URL/git-upload-pack HTTP/1.0
0032want 0a53e9ddeaddad63ad106860237bbf53411d11a7
0032have 441b40d833fdfa93eb2908e52742248faf0ee993
0000In this request, the client uses want and have lines to tell the server which objects it already owns locally and which ones it still needs. The final 0000 marks the end of the protocol stream and tells the server it can start sending object data. After receiving the delta information, the server compresses the objects the client needs and returns them in the HTTP response as an encoded pack stream.
The client then receives the required objects, unpacks them into its local object database, and finally checks out the working tree based on the tree object referenced by the latest commit.
For the challenge platform, the custom Git implementation mainly focuses on supporting the outer HTTP layer. The platform extracts the Git protocol payload from the HTTP request, writes that byte stream into the Git process, and then writes the process’s binary output back into the HTTP response. Which optional Git capabilities are actually available depends on the Git version installed on the server.
Once the Git transport side was done, the next step was checking the current version of the repository out into a working directory so the CI/CD subsystem could continue from there.
The HEAD file in a Git repository points to the latest commit, and from there you can obtain the corresponding tree and reconstruct the working directory from the objects referenced by that tree:
pub fn checkout_head(&self, dst_path: impl AsRef<Path>)
-> anyhow::Result<()> {
let dst_path = dst_path.as_ref();
let git_path = self.path.clone();
let mut index = gix_index::File::at(
git_path.join("index"),
Sha1,
Default::default()
)?;
let odb = gix::odb::at(git_path.join("objects"))?
.into_inner()
.into_arc()?;
let _outcome = gix_worktree::checkout(
&mut index,
dst_path,
move |oid, buf| odb.find_blob(oid, buf),
&mut progress::Discard,
&mut progress::Discard,
&AtomicBool::default(),
gix_worktree::checkout::Options {
overwrite_existing: true,
..Default::default()
},
)?;
Ok(())
}For Git operations, I chose the gitoxide stack to inspect the commit pointed to by HEAD and restore the full working tree into dst_path. One big reason was that it is pure Rust. Personally, I am very willing to spend a little extra effort reducing binary dependencies when I can.
Continuous integration and deployment
Once the Git storage module was in place, the next step was pairing it with a CI/CD module so challenge storage and publishing could work together. Challenge builds can take a long time, so it would be a bad idea to perform them directly inside an HTTP request handler. On top of that, builds can consume a lot of server resources, which means the system also needs a way to limit how much any given build can eat.
The design I went with used Redis queues to process build requests. When a challenge author asks the platform to build a challenge, that request is pushed into a Redis message queue. When the server starts, it spins up a dedicated thread that keeps listening on the queue. If a new build request appears, the worker pulls it out, opens the checker implementation for the corresponding challenge type, and then runs the build code against the repository contents. Based on the configuration written by the challenge author, it builds the required artifacts, containers, and other components, then stores the result in a stable directory for later use. Once that build is done, the worker returns to listening for the next request.
That way, server resource consumption stays bounded to the scope of a single challenge build. The platform avoids situations where too many simultaneous build requests exhaust the machine and leave the service unable to respond to users.
The build worker looks roughly like this:
pub async fn start_build_worker(mut cache: BuilderCache)
-> anyhow::Result<()> {
tokio::spawn(async move {
loop {
let challenge = cache.get_task().await?;
let mut checker = open_checker(&challenge).await?;
match checker.build().await {
Ok(_) => {
debug!("challenge built: {}:{}", challenge.id, challenge.name);
}
Err(err) => {
error!("failed to build challenge: {}", err);
continue;
}
}
}
});
Ok(())
}The open_checker function constructs the right checker according to the challenge type, and then the corresponding build function performs the actual build. Different challenge types need different build procedures, so this is decoupled with a factory-style design. If you want to add a new challenge type, you only need to implement the expected trait.
Here is a simple example of a build function for a static attachment challenge:
async fn build(&mut self) -> anyhow::Result<()> {
self.bucket.working.clean().await?;
self.bucket.checkout_to_working().await?;
self.bucket.working.lock().await?;
let base_path = self.bucket.working.path();
let config_file = base_path.join("config.toml");
let config = read_config(&config_file)?;
let mut check_flag = true;
for file in config.provided {
if !base_path.join(file).exists() {
check_flag = false;
break;
}
}
self.bucket.working.unlock().await?;
if check_flag {
self.bucket.stabilize().await?;
}
Ok(())
}Since a static attachment challenge only needs to verify that the files provided to players are correct, this is already enough.