Adding multi-camera for the side grayscale fisheye lens and multi-user training support on the Aria glasses #2932
Conversation
AntonioMacaronio
changed the title
AntonioMacaronio
force-pushed
the
aria-side-camera-support
branch
from
11f03c5
to
3a539ac
Compare
…xel_sampler.py, nerfacto_field.py have changes
…with aria multi-cam data
AntonioMacaronio
changed the title
AntonioMacaronio
changed the title
…callibration parameters
| input_image: np.ndarray, | ||
| input_calib: calibration.CameraCalibration, | ||
| output_focal_length: int, | ||
| ) -> [np.ndarray, calibration.CameraCalibration]: |
There was a problem hiding this comment.
is this a valid annotation? should it be Tuple[np.ndarray, calibration.CameraCalibration]?
|
|
||
| return output_image, updated_calib | ||
|
|
||
| def generate_circular_mask(numRows: int, numCols: int, radius: float): |
There was a problem hiding this comment.
return annotation would be nice!
| @@ -76,11 +78,10 @@ class TimedPoses: | |||
| t_world_devices: List[SE3] | |||
|
|
|||
|
|
|||
| def get_camera_calibs(provider: VrsDataProvider) -> Dict[str, AriaCameraCalibration]: | |||
| def get_camera_calibs(provider: VrsDataProvider, name="camera-rgb") -> AriaCameraCalibration: | |||
There was a problem hiding this comment.
super nit: can we annotate the name with a Literal type? (also below)
| if pinhole: | ||
| f_length = 500 if camera_name == "camera-rgb" else 170 | ||
| image_array, src_calib = undistort_image_and_calibration(image_array, src_calib, f_length) | ||
| intrinsic = [f_length, f_length, image_array.shape[1] // 2, image_array.shape[0] // 2] |
There was a problem hiding this comment.
nit: can we use tuples instead if things don't need to be mutated?
| @@ -68,6 +69,7 @@ class AriaImageFrame: | |||
| file_path: str | |||
| t_world_camera: SE3 | |||
| timestamp_ns: float | |||
| pinhole_intrinsic: List[int] | |||
There was a problem hiding this comment.
maybe tuple[float, float, float, float]?
| """Path to the second VRS file if provided""" | ||
| mps_data_dir2: Optional[Path] = None | ||
| """Path to the second MPS attachments if provided""" | ||
| points_file2: Optional[Path] = None |
There was a problem hiding this comment.
it feels a little weird to me to hardcode things to only support 1 or 2 VRS files, instead of being able to do multiple
would it make sense to replace the original vrs_file: Path with vrs_file: Tuple[Path, ...]? as well as mps_data_dir with mps_data_dir: Tuple[Path, ...]?
| @@ -184,59 +292,126 @@ class ProcessProjectAria: | |||
| """Path to Project Aria Machine Perception Services (MPS) attachments.""" | |||
| output_dir: Path | |||
| """Path to the output directory.""" | |||
| points_file: Optional[Path] = None | |||
| """Path to the point cloud file (usually called semidense_points.csv.gz) if not in the mps_data_dir""" | |||
There was a problem hiding this comment.
just wondering, when is the point cloud file to not in the MPS dir?
There was a problem hiding this comment.
When I downloaded the Aria Everyday Activities Dataset, the pointcloud was in its own separate folder so i went ahead and added support for both cases
There was a problem hiding this comment.
is it one shared point cloud across all users?
There was a problem hiding this comment.
actually it's multiple point cloud files, so rn main() will combine all of them since they should be using the same point of reference
…, also added more annotations
…Macaronio/nerfstudio into aria-side-camera-support
AntonioMacaronio
changed the title
Background
process_project_aria.pyonly works for the main camera, and only supports 1 user. Since the Aria glasses can sync multiple users by timestamp with the same point of reference for MPS, it is a nice functionality to support.Changes