Guide for how to build, configure, and develop SymForce.
SymForce aims to follow Python standards. The core
symforce package lives in the equivalently named subdirectory at the top level. Tests, documentation, etc live at the top level outside of the core package.
symforce add the top level to the Python path.
See the module reference for the core package structure.
SymForce is primarily written in Python and C++, and is Python 3.8+ and C++14 compatible. The build system is CMake for the C++ components, and optionally pip / setuptools on top for Python packaging. See the Build section on the Homepage for build instructions.
Additional useful commands#
SymForce also has a top level Makefile which is not used by the build, but provides some high level commands for development:
Run tests which update (most) generated code
Run tests which update all generated code
Run tests and open coverage report
Build docs + open in browser
Run the code formatter (black, clang-format)
Check types with mypy
Check formatting and types
This documentation is built with Sphinx, including automatic parsing of the code to generate a module reference using sphinx-apidoc. The code uses Google Style docstrings to annotate all modules, classes, and functions. Docs pages are
.rst files in
docs, and the Sphinx config file is
SymForce uses the logging module. You can import and use the logger like this:
>>> from symforce import logger >>> logger.warning('houston, we have a problem') codegen_test.test_codegen_cpp():126 WARNING -- houston, we have a problem
You can configure the log level using
symforce.set_log_level() or by setting the
SYMFORCE_LOGLEVEL environment variable. The default is
Testing and Coverage#
SymForce is heavily tested, targeting close to 100% code coverage.
Tests live in
test and use unittest. Additionally, coverage.py is used to run tests while measuring code coverage. The generated coverage report also provides a great view into what methods need to be tested and what code is potentially unused.
SYMFORCE_LOGLEVEL=DEBUG python test/symforce_codegen_test.py
cd build; ctest
ls test/*_test.py | xargs -n1 -P $(nproc) python
When debugging a specific test, the use of ipdb is highly recommended, as is reproducing the most minimal example of the issue in a notebook.
Symforce uses the Black formatter for Python code. To quote the authors:
Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from nagging about formatting. You will save time and mental energy for more important matters.
make format will format the entire codebase. It’s recommended to develop with VSCode and integrate black.
Much of the core functionality of SymForce is in generating code using the Jinja template language. It’s relatively simple and easy to use - you pass it a template file in any language and a python dictionary of data, and it spits out the rendered code.
For example template files, see symforce/codegen/backends/cpp/templates.
SymForce uses the SymPy API, but supports two implementations of it. The SymPy implementation is pure Python, whereas the SymEngine implementation is wrapped C++. It can be 100-200 times faster for many operations, but is less fully featured and requires a C++ build.
To set the symbolic API, you can either use
symforce.set_symbolic_api() before any other imports, or use the
SYMFORCE_SYMBOLIC_API environment variable with the options
symengine. By default SymEngine will be used if found, otherwise SymPy.
You should be able to build Python wheels of symforce the standard ways. We recommend using
build, i.e. running
python3 -m build --wheel from the
symforce directory. By default,
this will build a wheel that includes local dependencies on the
packages (which are separate Python packages from
symforce itself). For distribution, you’ll
typically want to set the environment variable
building, and also run
python3 -m build --wheel third_party/skymarshal and
python3 -m build --wheel gen/python to build wheels for those packages separately.
For SymForce releases, all of this is handled by the
build_wheels GitHub Actions workflow. This
workflow is currently run manually on a commit, and produces a
symforce-wheels.zip artifact with
wheels (and sdists) for distribution (e.g. on PyPI). It doesn’t upload them to PyPI - to do that
(after verifying that the built wheels work as expected) you should download and unzip the archive,
and upload to PyPI with
python -m twine upload [--repository testpypi] --verbose *.
Adding new types#
To add a new geo or cam type to SymForce:
For geo types, you should add it to the
notebooks/tangent_D_storage.ipynbnotebooks, and use the results there for your symbolic implementation.
Create a test of your symbolic type, for example
For geo types, register their numerical equivalents in
Add any custom methods you’d like on the runtime numerical classes to the corresponding file in the
custom_methodsdirectory for each backend language
For geo types, add them to the
"Test implicit construction"and
"Test lie group ops"test cases in