Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry & Saved has just 2 files #825

Open
cleanev opened this issue Mar 4, 2024 · 60 comments
Open

Sentry & Saved has just 2 files #825

cleanev opened this issue Mar 4, 2024 · 60 comments

Comments

@cleanev
Copy link

cleanev commented Mar 4, 2024

Describe the problem

Till February 23, all Sentry & Saved filed were synced using rclone. 2/23 at the end of the day updated to latest version 2024.2.7 and now every single day there are just 2 files named event.json and thumb.png files are the only 2 that gets uploaded. What else should I look into. Will check another car where 2024.2.7 version got updated few days later.

Today (March3 ) updated teslausb using self update and still cannot get older saved and sentry files are uploaded.
Tried to check to best of my understanding and found sentry_found_files file in tmp folder that has below logged

SentryClips/2024-02-23_12-12-30/2024-02-23_12-01-03-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-01-03-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-01-03-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-02-04-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-02-04-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-02-04-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-02-04-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-03-04-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-03-04-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-03-04-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-03-04-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-04-05-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-04-05-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-04-05-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-04-05-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-05-05-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-05-05-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-05-05-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-05-05-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-06-06-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-06-06-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-06-06-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-06-06-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-07-06-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-07-06-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-07-06-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-07-06-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-08-07-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-08-07-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-08-07-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-08-07-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-09-07-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-09-07-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-09-07-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-09-07-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-11-08-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-11-08-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-11-08-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-11-08-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-12-09-back.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-12-09-front.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-12-09-left_repeater.mp4
SentryClips/2024-02-23_12-12-30/2024-02-23_12-12-09-right_repeater.mp4
SentryClips/2024-02-23_12-12-30/event.json
SentryClips/2024-02-23_12-12-30/thumb.png
SentryClips/2024-02-24_10-50-05/event.json
SentryClips/2024-02-24_10-50-05/thumb.png
SentryClips/2024-02-24_11-11-09/event.json
SentryClips/2024-02-24_11-11-09/thumb.png
SentryClips/2024-02-26_11-52-07/event.json
SentryClips/2024-02-26_11-52-07/thumb.png
SentryClips/2024-02-26_18-36-41/event.json
SentryClips/2024-02-26_18-36-41/thumb.png
SentryClips/2024-02-26_19-13-32/event.json
SentryClips/2024-02-26_19-13-32/thumb.png
SentryClips/2024-02-28_13-24-50/event.json
SentryClips/2024-02-28_13-24-50/thumb.png
SentryClips/2024-02-28_15-11-59/event.json
SentryClips/2024-02-28_15-11-59/thumb.png
SentryClips/2024-02-28_18-00-21/event.json
SentryClips/2024-02-28_18-00-21/thumb.png
SentryClips/2024-02-28_19-18-50/event.json
SentryClips/2024-02-28_19-18-50/thumb.png
SentryClips/2024-03-02_19-35-31/event.json
SentryClips/2024-03-02_19-35-31/thumb.png
SentryClips/2024-03-02_20-09-48/event.json
SentryClips/2024-03-02_20-09-48/thumb.png

Below is what shows up in archiveloop.log right after rebooting

==============================================
Sun  3 Mar 21:20:02 EST 2024: Starting archiveloop at 43.21 seconds uptime...
Sun  3 Mar 21:20:03 EST 2024: Running fsck on /backingfiles/cam_disk.bin...
Sun  3 Mar 21:20:04 EST 2024: | fsck from util-linux 2.33.1
Sun  3 Mar 21:20:08 EST 2024: | fsck.fat 4.1 (2017-01-24)
Sun  3 Mar 21:20:08 EST 2024: | Long filename fragment "2023-10-20_20" found outside a LFN sequence.
Sun  3 Mar 21:20:08 EST 2024: |   (Maybe the start bit is missing on the last fragment)
Sun  3 Mar 21:20:08 EST 2024: |   Not auto-correcting this.
Sun  3 Mar 21:20:08 EST 2024: | Long filename fragment "2023-05-31_16" found outside a LFN sequence.
Sun  3 Mar 21:20:08 EST 2024: |   (Maybe the start bit is missing on the last fragment)
Sun  3 Mar 21:20:08 EST 2024: |   Not auto-correcting this.
Sun  3 Mar 21:20:08 EST 2024: | Long filename fragment "2023-04-16_14" found outside a LFN sequence.
Sun  3 Mar 21:20:08 EST 2024: |   (Maybe the start bit is missing on the last fragment)
Sun  3 Mar 21:20:08 EST 2024: |   Not auto-correcting this.
Sun  3 Mar 21:20:08 EST 2024: | Long filename fragment "2023-04-09_18" found outside a LFN sequence.
Sun  3 Mar 21:20:08 EST 2024: |   (Maybe the start bit is missing on the last fragment)
Sun  3 Mar 21:20:08 EST 2024: |   Not auto-correcting this.
Sun  3 Mar 21:20:08 EST 2024: | Wrong checksum for long file name "2024-02-23_15-12-44-back.mp4".
Sun  3 Mar 21:20:08 EST 2024: |   (Short name 209B50~1.MP4 may have changed without updating the long name)
Sun  3 Mar 21:20:08 EST 2024: |   Not auto-correcting this.
Sun  3 Mar 21:20:08 EST 2024: | /dev/loop0p1: 423 files, 742871/1965055 clusters
Sun  3 Mar 21:20:08 EST 2024: Finished fsck on /backingfiles/cam_disk.bin.
Sun  3 Mar 21:20:08 EST 2024: taking snapshot of cam disk in /backingfiles/snapshots/snap-003216
Sun  3 Mar 21:20:09 EST 2024: waiting for autofs to be active
Sun  3 Mar 21:20:10 EST 2024: waiting for autofs to be active
Sun  3 Mar 21:20:12 EST 2024: waiting for autofs to be active
Sun  3 Mar 21:20:13 EST 2024: waiting for autofs to be active
Sun  3 Mar 21:20:14 EST 2024: waiting for autofs to be active
Sun  3 Mar 21:20:16 EST 2024: took snapshot
Sun  3 Mar 21:20:19 EST 2024: comparing new snapshot with /backingfiles/snapshots/snap-003215/snap.bin
Sun  3 Mar 21:20:19 EST 2024: new snapshot is identical to previous one, discarding
Sun  3 Mar 21:20:31 EST 2024: Trying to set time...
Sun  3 Mar 21:21:21 EST 2024: Time adjusted by 50.088968 seconds after 0.190000 seconds
Sun  3 Mar 21:21:21 EST 2024: Not keeping car awake.
Sun  3 Mar 21:21:21 EST 2024: Archiving...
Sun  3 Mar 21:21:22 EST 2024: Checking saved folder count...
Sun  3 Mar 21:21:22 EST 2024: There are 0 event folder(s) with 0 file(s) and 0 track mode file(s) to move.  0 short recording(s) will be skipped.
Sun  3 Mar 21:21:23 EST 2024: Ensuring cam file is mounted...
Sun  3 Mar 21:21:23 EST 2024: Disconnecting usb from host...
Sun  3 Mar 21:21:23 EST 2024: Disconnected usb from host.
Sun  3 Mar 21:21:23 EST 2024: Mounting /mnt/cam...
Sun  3 Mar 21:21:23 EST 2024: Mounted /mnt/cam.
Sun  3 Mar 21:21:23 EST 2024: Ensured cam file is mounted.
Sun  3 Mar 21:21:23 EST 2024: cleaning cam mount
Sun  3 Mar 21:21:24 EST 2024: done cleaning cam mount
Sun  3 Mar 21:21:25 EST 2024: Trimming free space in /mnt/cam, which has 1682 extents
Sun  3 Mar 21:21:25 EST 2024: Trim complete, image now has 1682 extents
Sun  3 Mar 21:21:25 EST 2024: Unmounting /mnt/cam...
Sun  3 Mar 21:21:25 EST 2024: Unmounted /mnt/cam.
Sun  3 Mar 21:21:25 EST 2024: Finished archiving.
Sun  3 Mar 21:21:25 EST 2024: Music archive not configured or unreachable
Sun  3 Mar 21:21:25 EST 2024: Connecting usb to host...
Sun  3 Mar 21:21:26 EST 2024: Connected usb to host.
Sun  3 Mar 21:21:31 EST 2024: Waiting for archive to be unreachable...


### Device

Raspberry Pi Zero W

### OS Image

Prebuilt TeslaUSB image

### Car Model

Model Y

### USB connection

Center console

### Logs

_No response_

### Additional information

_No response_
@marcone
Copy link
Owner

marcone commented Mar 5, 2024

I have seen this issue too, as have others. Since it only started happening recently I suspect it's due to a Tesla software update, but it's unclear what exactly the trigger is. If I clear out the "cam" drive, it starts working properly again.

@cleanev
Copy link
Author

cleanev commented Mar 5, 2024

Thanks for confirming @marcone. I checked 2018 Model 3 and that does exactly the same.
When you say clearing out “cam” drive is it the Teslacam folder on SD card? I will definitely give that a try once you confirm to see if it works again like you observed.

@marcone
Copy link
Owner

marcone commented Mar 5, 2024

Connecting the Pi to a PC and deleting the content of the TeslaCam folder in the "CAM" drive is what I did. Don't delete the TeslaCam folder itself.
Or if you want to do it without taking the Pi out of the car:

sudo -i
bin/disable_gadget.sh
mount /mnt/cam
rm -rf /mnt/cam/TeslaCam/*
umount /mnt/cam
bin/enable_gadget.sh

This probably isn't a permanent fix, but it made it work again for me for now.

@brswattt
Copy link

brswattt commented Mar 6, 2024

Connecting the Pi to a PC and deleting the content of the TeslaCam folder in the "CAM" drive is what I did. Don't delete the TeslaCam folder itself. Or if you want to do it without taking the Pi out of the car:

sudo -i
bin/disable_gadget.sh
mount /mnt/cam
rm -rf /mnt/cam/TeslaCam/*
umount /mnt/cam
bin/enable_gadget.sh

This probably isn't a permanent fix, but it made it work again for me for now.

I believe this fixed it for me, will verify when I get home from work.

@Gizmotoy
Copy link

Gizmotoy commented Mar 7, 2024

Having the same issue here. I gave the above series of "in-vehicle" commands a shot. There is one modification needed. I got an error that I couldn't rm on a read-only file. The correct sequence, I think, is:

sudo -i
bin/disable_gadget.sh
mount -o rw /mnt/cam
rm -rf /mnt/cam/TeslaCam/*
umount /mnt/cam
bin/enable_gadget.sh

I'll monitor and report back if this fixes it.

@marcone
Copy link
Owner

marcone commented Mar 7, 2024

you shouldn't need to add that -o rw to the mount command. It should be mounted read-write by default. If it's not, there's probably some kind of filesystem corruption.

@cleanev
Copy link
Author

cleanev commented Mar 8, 2024

Connecting the Pi to a PC and deleting the content of the TeslaCam folder in the "CAM" drive is what I did. Don't delete the TeslaCam folder itself. Or if you want to do it without taking the Pi out of the car:

sudo -i
bin/disable_gadget.sh
mount /mnt/cam
rm -rf /mnt/cam/TeslaCam/*
umount /mnt/cam
bin/enable_gadget.sh

This probably isn't a permanent fix, but it made it work again for me for now.

Thanks @marcone - it works for now on 2 separate instances.

@coolham123
Copy link

I'm having a similar issue. TeslaUSB was only backing up 2 files. I ran the commands @marcone suggested. I am using TeslaFi to keep the car awake, and the archive log never gets past "Starting background task to keep car awake.". I verified through the TeslaFi API page, the call is going through and is received by the vehicle. I re-rolled the API token for good measure and updated the config, restarted, aswell as ran the update script to make sure I was on the latest TeslaUSB version. The Diagnostics page (attached) does not immediately jump out at me.

teslausb_diagnostics.txt

@Gizmotoy
Copy link

Gizmotoy commented Mar 9, 2024

This worked for me for a few days and now it’s exhibiting the behavior again. Notably on one new vehicle (2024) which got a new and freshly-provisioned Pi Zero W teslausb installation. Now I’m also seeing the same on an existing installation on a 2020 vehicle where teslausb had been working flawlessly for years.

Seems related to a recent vehicle SW update, I guess? Both are running v11.1 ⁦(2024.2.7 d522c44937f7)⁩.

If it's not, there's probably some kind of filesystem corruption.

What’s the recommended way to check for/repair this for these mounts?

@cleanev
Copy link
Author

cleanev commented Mar 15, 2024

Broke after few days again on 2 cars. Just 2 files.
@Gizmotoy - this seems to be Tesla firmware update, both vehicles are on 2024.2.7 - 2018 RWD 3 and 2020 Y.
Maybe wait till next Tesla update.

@Gizmotoy
Copy link

Gizmotoy commented Mar 15, 2024

Mine are also again broken.

this seems to be Tesla firmware update

That's the big concern, I think. It doesn't seem super likely to me that Tesla will make fixes for this use case (or even notice they've broken it).

@marcone , is there anything we could do to help provide some insight into what is actually going on here? Run a diagnostic build? Capture logs? Happy to help investigate.

@brswattt
Copy link

Same here, broke after a while. Going back to manual USB until this is resolved, if it can be. I'll format it and see if theres any obvious directory changes. I'll be happy to help as well. What exactly has changed? How the video files are saved? Naming?

@marcone
Copy link
Owner

marcone commented Mar 15, 2024

@marcone , is there anything we could do to help provide some insight into what is actually going on here? Run a diagnostic build? Capture logs? Happy to help investigate.

I don't think there is anything that can be done on the TeslaUSB side. I think this is just a bug on Tesla's side, since it started recently with no notable TeslaUSB changes, and I'm seeing problems even with a regular USB drive.

@brswattt
Copy link

@marcone , is there anything we could do to help provide some insight into what is actually going on here? Run a diagnostic build? Capture logs? Happy to help investigate.

I don't think there is anything that can be done on the TeslaUSB side. I think this is just a bug on Tesla's side, since it started recently with no notable TeslaUSB changes, and I'm seeing problems even with a regular USB drive.

So you think this problem is widespread to all Tesla dashcam users regardless if they are using teslausb or not? What problems do you see with the regular usb drive?

@Gizmotoy
Copy link

I was wondering the same. I flipped my 2024 back to a USB drive and it seems to be working fine so far, while the 2020 continues to fail after clearing the TeslaCam folder every few days. I can't say I see any complaints of this on the various Tesla forums despite scouring pretty hard, but it doesn't mean it's not happening.

@daffster
Copy link

I was seeing this problem, and tried every fix in this thread. I still was only seeing 2 files per "event" and nothing thereafter.

I've gone back to using an SSD plugged directly into the car (2022 M3) and its recording just fine.

@marcone
Copy link
Owner

marcone commented Mar 18, 2024

@Gizmotoy are you using the center console or glove box? I've personally seen it on a Model Y with center console USB, and @cleanev also reported using the center console.
With my Model Y, the problem started out as only having those 2 files in the sentry folder as described above, but has since evolved to the car simply not seeing the USB drive at all until the car is reset, at which point it'll work again for a few days.

@Gizmotoy
Copy link

I have one of each: my 2020 doesn’t have a glove box port at all. I still have that TeslaUSB Pi Zero W device installed and it fails every few days and I do the process above to resolve it temporarily.

My 2024 has the glove box port and TeslaUSB was installed there. Subjectively, the behavior was worse in the 2024, lasting only a day or two max, so I replaced it with the original USB drive. It’s been solid again and I’ve encountered no issues on USB at all.

@brswattt
Copy link

I have a 2023 Model 3 RWD with the USB plugged into the glovebox and have had no problems using a default USB stick. Same behavior after a couple days with the fix above on teslausb.

@cleanev
Copy link
Author

cleanev commented Mar 19, 2024

The consensus reading these comments is Tesla made a change affecting TeslaUSB in all models, regardless of where you plug in while allow plain vanilla USB key to work.
Hope is that if one of us gets the latest version 2024.8.4 that is currently being deployed, we will get to know if TeslaUSB works or not.

@Gizmotoy
Copy link

My 2020 just received 2024.8.4. Installing now. It has TeslaUSB still installed. Hope to understand if this is just a thing they've fixed within the next few days.

@andrewvo88
Copy link

This happened twice for me. What I've notice is that the issue occurred after a software update. I updated to 2024.8.4 and the issue happened again. I'll keep an eye out but that is something I notice for me.

@cleanev
Copy link
Author

cleanev commented Apr 10, 2024

Not sure if anyone other than @mondagoar has had any success. Model Y-2020 with 2024.3.10, Model 3 2018 with 2024.2.7 still has same issue where after removing the files it lasts for a few drives with and then 2 files again. Something that Tesla did or does reinstalling take care of this?

@Gizmotoy
Copy link

I tried reinstalling and was still affected. Running 2024.3.10 and 2024.8.7.

I switched back to flash drives until this is sorted. I was almost hit and had to swerve off the road last week and teslausb didn’t capture anything but the two meta files. Flash is safer for now, I guess.

@ukoda
Copy link

ukoda commented Apr 11, 2024

@Gizmotoy Have you verified that the camera footage is actually present on the USB drive? Before finding this issue I tried switching to a USB flash drive and it did not have the content on it I expected. Is it possible the camera footage is saved to internal storage on the Tesla system?
I really hope this issue is resolved soon as really don't like not having camera footage available if there was an accident.

@Gizmotoy
Copy link

I did pull the Raspberry Pi to confirm and no, the video files were indeed missing.

I’ve had no problems on flash drives on either vehicle, so switched back to those for now.

@marcone
Copy link
Owner

marcone commented Apr 11, 2024

I did pull the Raspberry Pi to confirm and no, the video files were indeed missing.

Were there RecentClips though? What I noticed when this issue first started happening, RecentClips were still being recorded and snapshotted, and so it was still possible to see Sentry events in the RecentClips timeline in the web interface. It was like the car just stopped bothering to move Sentry recordings to their own folder.

Then with a later software update the car stopped recording altogether. Or rather it stopped seeing the drive, including plain USB drives.

@ukoda
Copy link

ukoda commented Apr 11, 2024

@marcone That is inline with what I saw, plain USB drives had the same issue. @Gizmotoy You mention checking the RPi and seeing the same thing as others but what about the USB drive you are now putting your trust in? I would seriously suggest you remove your USB drive and check it on the PC too.
My gut feel is an update to the Tesla software has stop saving video to external storage in favor of internal storage. If that is the case then there is no fix that can be done on the RPi and we just have to wait for Tesla to re-enable external storage.

@Gizmotoy
Copy link

Gizmotoy commented Apr 11, 2024

USB drive has been 100% flawless for me. We discussed this pretty extensively above over the past month or two. The behavior has followed through 3 or 4 different vehicle software versions too. Whatever it is is only affecting Teslausb for me.

There might have been RecentClips. I’ve done the posted recovery process dozens of times so I didn’t think too much of it when I had to do it again, but then I thought about it a bit more and just put the flash drive back in instead. I guess a mistake.

@cleanev
Copy link
Author

cleanev commented Apr 14, 2024

I did pull the Raspberry Pi to confirm and no, the video files were indeed missing.

Were there RecentClips though? What I noticed when this issue first started happening, RecentClips were still being recorded and snapshotted, and so it was still possible to see Sentry events in the RecentClips timeline in the web interface. It was like the car just stopped bothering to move Sentry recordings to their own folder.

Then with a later software update the car stopped recording altogether. Or rather it stopped seeing the drive, including plain USB drives.

@marcone - yes RecentClips does have associated clips for specific time - say the folder has date-time then I can find all the files around this time and many more from earlier & later in RecentClips folder. As you stated this may be something that the car stopped copying over all MP4 files to its respective folders while still copying thumbs.db & events files.

reading @tomrund post, I see 4 items when FSCKing related to LFN. Also the snapshot number was up to 3578 since I built this Pi.

What I did next is below & while it seems to work please
DO NOT ATTEMPT if you are not comfortable as this is not a tried solution

I got a bit brave and removed cam_disk.bin from backingfiles folder while logged in as sudo -i.
Ended up with recurring errors in achiveloop.log.
To remediate this I ran setup-teslausb from /root/bin/ folder with upgrade option.
Somehow doing what I did there are files that are no longer accessible in /var/www/html/RecentFiles or /SavedFiles or SentryFiles folder, like so when I list directory. New directories since yesterday created using test script have all the files. Will need to figure out how to clean up so they are no longer showing in browser - if clicked on folder in browser, 500 Internal Server error pops up.

ls: cannot access '2024-04-12/2024-04-12_23-41-09-back.mp4': No such file or directory
ls: cannot access '2024-04-12/2024-04-12_23-41-09-front.mp4': No such file or directory
ls: cannot access '2024-04-12/2024-04-12_23-41-09-left_repeater.mp4': No such file or directory
ls: cannot access '2024-04-12/2024-04-12_23-41-09-right_repeater.mp4': No such file or directory
ls: cannot access '2024-04-12/2024-04-12_23-42-09-back.mp4': No such file or directory
ls: cannot access '2024-04-12/2024-04-12_23-42-09-front.mp4': No such file or directory

Above cleared out all the snapshots & now started with 0000. Also cleared archiveloop.log as it had grown to 4MB, not that it matters in 256GB microSD.
Used @marcone createtestfiles script and adapted to create sentry and saved files.
Seems to be working well & has worked for 20+ I created test files; so am hopeful that it will work when plugged in the car, but only time will tell. I will report later once I see success or lack thereof.

Best

@Zingabopp
Copy link

Anyone having this problem using exFAT? It's been good for a few days since I switched, could just be luck though. On FAT32 mine was also recording RecentClips fine, but failed to move the files into the SavedClips/SentryClips folders.

@brswattt
Copy link

brswattt commented Apr 15, 2024 via email

@cleanev
Copy link
Author

cleanev commented Apr 15, 2024

Is your backingfiles formatted as ExFAT? My installation is XFS and all others are shown below on PiZeroW

Filesystem Type Size Used Avail Use% Mounted on
/dev/root ext4 2.0G 1.4G 462M 75% /
devtmpfs devtmpfs 183M 0 183M 0% /dev
tmpfs tmpfs 216M 0 216M 0% /dev/shm
tmpfs tmpfs 216M 5.8M 210M 3% /run
tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs tmpfs 216M 0 216M 0% /sys/fs/cgroup
tmpfs tmpfs 216M 0 216M 0% /var/lib/nginx
tmpfs tmpfs 216M 0 216M 0% /var/spool
tmpfs tmpfs 216M 0 216M 0% /var/tmp
tmpfs tmpfs 216M 364K 215M 1% /tmp
tmpfs tmpfs 216M 4.0K 216M 1% /var/lib/ntp
tmpfs tmpfs 216M 0 216M 0% /var/log/nginx
/dev/mmcblk0p3 xfs 237G 8.9G 228G 4% /backingfiles
/dev/mmcblk0p1 vfat 247M 51M 197M 21% /boot
/dev/mmcblk0p4 ext4 93M 3.0M 83M 4% /mutable
tmpfs tmpfs 43M 0 43M 0% /run/user/1000

@Zingabopp
Copy link

No, I believe that's still XFS. All I do is uncomment export USE_EXFAT=true in teslausb_setup_variables.conf


which, if I'm interpreting things correctly, is for the internal file system of the bin files (such as cam_disk.bin). Formatting the 'drive' from the Tesla menu may have the same effect, haven't tried that though.

@Gizmotoy
Copy link

Gizmotoy commented Apr 16, 2024

An interesting lead. The two flash drives I've been using that have been 100% reliable are both ExFAT. It looks like my teslausb instances are both Fat32.

I wonder if this would explain why we're seeing different reports on the behavior of USB flash drives. Were those of you who were also having problems with flash drives using Fat32-formatted flash drives?

Edit:
Per the above, the full config setting

# Uncomment to use ExFAT filesystem instead of FAT32.
# ExFAT is the filesystem used by Tesla when the drive is formatted in the car.
# Using ExFAT is not recommended for the TeslaUSB prebuilt image, as it is
# based on an older kernel that has no TRIM support for ExFAT.
export USE_EXFAT=true

Could be rough to lose TRIM for a use case like this. According to Wikipedia this feature was added in 5.13. The prebuilt image appears to use 5.10.63+, based on my install, so the comment appears to still be applicable. This configuration would lack TRIM support.

I replaced my teslausb FAT32 formatted drive with an exFAT one despite the loss of TRIM just to see if this works to fix the problem.

@Zingabopp
Copy link

The pre-release Bullseye image is on 6.1 and supports TRIM for exFAT, although it does have an annoying issue with my Pi Zero 2 W where the wifi driver crashes when it leaves the range of my AP.

@Gizmotoy
Copy link

I tried the pre-release image in my vehicles. No issues yet with exFAT storing sentry mode clips properly. I do have some trouble getting the drive recognized, though. It seems like the car doesn't recognize the drive every third time or so (works fine on my computer, and works in the car after I reboot the center display).

@tomrund
Copy link

tomrund commented Apr 19, 2024

Hey, I don’t Alan’s to rain on anyone’s parade. But I was using the default file system for years before it acted up. So my assumption would be that any reformat, regardless of the chosen format, would seem to resolve the issue.

@tomrund
Copy link

tomrund commented Apr 19, 2024

Also if the GitHub gui wasn’t trash on mobile, I’d happily fix my typo from above.

@nickastaldo
Copy link

Are you saying that formatting and reinstalling the TeslaUSB image would fix this issue? I've kept my current install in the hopes that there was an eventual tweak or fix for this.

@tomrund
Copy link

tomrund commented Apr 20, 2024

Yes, I did a fresh install, but I believe just removing and recreating the cam drive backing file would likely also be sufficient. After my fresh install things have been working normally again for a couple weeks without changing any settings or hardware.

@nickastaldo
Copy link

Yes, I did a fresh install, but I believe just removing and recreating the cam drive backing file would likely also be sufficient. After my fresh install things have been working normally again for a couple weeks without changing any settings or hardware.

Sweet, just reformatted the drive and everything is working again. I have no idea why it did that, but I'm happy it's working again. Appreciate it, tomrund!

@rworne
Copy link

rworne commented Apr 27, 2024

The pre-release Bullseye image is on 6.1 and supports TRIM for exFAT, although it does have an annoying issue with my Pi Zero 2 W where the wifi driver crashes when it leaves the range of my AP.

I had this issue with WiFi crapping out on my RP4 with the Bullseye image, I found this comment that so far, seems to work for me:

Add "modules-load=brcmfmac roamoff=1 feature_disable=0x82000" into /boot/cmdline.txt

See here for more info:
https://retropie.org.uk/forum/topic/32462/can-not-connect-to-wifi/5
and here:
https://forums.raspberrypi.com/viewtopic.php?t=349845

Ultimately I had to reinstall to solve this issue (sentry folders have no videos). Funny enough the car seems to see them just fine. I di have a growing list of folders with weird names that seemed to correspond to the number of sentry events, but I didn't look into it that much. So while recent clips would grow, I could command the car to erase the video files or reformat the drive and when I checked it again, all the files were still there. So I did a reinstall.

I made the drives ExFAT just in case and it is working. Then the issue above happened a few minutes after I got home and it started syncing. Now it stays up, so that's good. The only remaining issue is the script won't keep the car awake unless I manually clear the cache and do the dance mentioned in another issue.

@usuryadevara
Copy link

Connecting the Pi to a PC and deleting the content of the TeslaCam folder in the "CAM" drive is what I did. Don't delete the TeslaCam folder itself. Or if you want to do it without taking the Pi out of the car:

sudo -i
bin/disable_gadget.sh
mount /mnt/cam
rm -rf /mnt/cam/TeslaCam/*
umount /mnt/cam
bin/enable_gadget.sh

This probably isn't a permanent fix, but it made it work again for me for now.

This method doesn't work for me tried couple times. Still has only two files in folders like these:
SentryClips/2024-03-29_20-06-48/event.json
SentryClips/2024-03-29_20-06-48/thumb.png

@Gizmotoy
Copy link

Gizmotoy commented May 1, 2024

That process solves it pretty routinely for me, but the fix is temporary and returns.

Here’s the things I have tried that work temporarily, but the issue eventually returns:

  1. The series of commands above to delete the corrupted files
  2. Formatting the TeslaCam drive from the Tesla UI
  3. Formatting the TeslaCam drive from a computer
  4. Full wipe/reinstallation

Here’s what I know works:

  1. Tesla’s provided flash drive (ExFAT)

Here’s what I’m uncertain works:

  1. Formatting the cam drive as ExFAT via TeslaUSB config
  2. Wipe/reinstallation using the Bullseye pre-release image AND an ExFAT drive via config

On my vehicle, now running #2 from this list for about 2.5 weeks, I haven’t seen the issue recur. However, it introduces another issue for about 20% of trips where the drive is not recognized in the vehicle until resetting the MCU.

@marcone
Copy link
Owner

marcone commented May 1, 2024

This method doesn't work for me tried couple times. Still has only two files in folders like these:
SentryClips/2024-03-29_20-06-48/event.json
SentryClips/2024-03-29_20-06-48/thumb.png

Were you expecting these steps to fix existing Sentry events from over a month ago?

@usuryadevara
Copy link

This method doesn't work for me tried couple times. Still has only two files in folders like these:
SentryClips/2024-03-29_20-06-48/event.json
SentryClips/2024-03-29_20-06-48/thumb.png

Were you expecting these steps to fix existing Sentry events from over a month ago?

No i tried recently. Same for the files from 2024-04-29 and 2024-04-30. That method is not working.

@Zingabopp
Copy link

On my vehicle, now running #2 from this list for about 2.5 weeks, I haven’t seen the issue recur. However, it introduces another issue for about 20% of trips where the drive is not recognized in the vehicle until resetting the MCU.

I also had the drive not recognized issue a few weeks ago. I thought it had something to do with the Rock Pi 4C+. Every time I plugged that in I'd have to reset the MCU to even get the stock flash drive to be recognized. Issue went away after a software update, had it once today right after the 2024.3.25 update installed. Haven't had the missing video files issue since switching to exFAT.

@rworne
Copy link

rworne commented May 2, 2024

What I find odd about this is how there are only two files (and no videos) in the directories, yet while the TeslaUSB is plugged into the car, it can see and play the sentry and dashcam clips just fine?

@Zingabopp
Copy link

Mine wasn't able to play the clips at all. The clips are (or were) saved to the drive in RecentClips, it seems to be an issue with moving them into the event folder.

@jmilesri
Copy link

jmilesri commented May 3, 2024

For those running a RPi 4C+ what benefit(s) does it offer over a RPi 4 as it relates to Teslausb?

@ravibowman
Copy link

I use a RockPi 4C+, the benefit for me is USB 3.0 OTG so you dont have to see the slow USB connected notification all the time. My only issue with this SBC is I cant get the HotSpot feature to work. Everything else has been fine.

@miles267
Copy link

miles267 commented May 3, 2024

I use a RockPi 4C+, the benefit for me is USB 3.0 OTG so you dont have to see the slow USB connected notification all the time. My only issue with this SBC is I cant get the HotSpot feature to work. Everything else has been fine.

@ravibowman is it confirmed hat the USB 3.0 OTG is much faster than the RPi 4? While I reliably use a RPi 4, I've come to ignore the slow USB messages entirely yet there's never been a speed issue, at least thus far.

@tomrund
Copy link

tomrund commented May 3, 2024

Just to add another datapoint, after experiencing this issue for the first time, I re-imaged my TeslaUSB instance and had my first successful Sentry recording afterwards at 2024-04-02_22-34-18 and it worked all the way until 2024-04-24_19-08-52. At some point between that recording and a recording on 2024-04-26_13-48-31, things went awry and I started to experience this issue again.

These dates do not coincide with any car software updates (I received 2024.3.25 on Apr 29, 2024 and 2024.3.15 on Apr 12, 2024), I am using the default filesystem and haven't tried switching to exFAT, although I'm tempted to.

This time, rather than re-image, I just removed the cam drive backing file as I previously suggested and re-ran setup and things are working normally once again.

sudo -i && bin/disable_gadget.sh && rm -rf /backingfiles/cam_disk.bin && bin/setup-teslausb upgrade

I was prompted to Delete snapshots and recreate recording and music drives? (yes/cancel), I said yes, setup completed, the device rebooted, and I was back in action again.

@tomrund
Copy link

tomrund commented May 5, 2024

I ended up in this state again. I saw some discussions in the Discord where users were reporting fsck recovered files after mounting their cam_disk.bin, and after removing these files their issue went away but this hasn't been my experience. When I mount my cam_disk.bin there are no fsck recovered files anywhere in the filesystem.

I'm once again getting the Long filename fragment found outside a LFN sequence errors from fsck as I did previously when I encountered this issue, and as the original reporter of this issue was experiencing. I previously tried to repair these errors in a way that seemed to fix my cam_disk.bin but was unsuccessful, I'll keep looking for a solution.

@tomrund
Copy link

tomrund commented May 8, 2024

I believe this issue has been resolved with this commit. I'd recommend updating with sudo -i && bin/setup-teslausb upgrade when you have a chance to receive the fix.

@nickastaldo
Copy link

nickastaldo commented May 12, 2024

I believe this issue has been resolved with this commit. I'd recommend updating with sudo -i && bin/setup-teslausb upgrade when you have a chance to receive the fix.

Will this fix it if it stops working? Mine was working when I reinstalled everything from scratch, but now it reverted to saving only two small files. I did push the dashcam button on the screen to capture an event manually today.. I wonder if that triggers this problem?

@ukoda
Copy link

ukoda commented May 12, 2024

@nickastaldo I did an update then checked the change in the commit has been applied. It hadn't, so I simply manually edited the file in question to match the commit. Only limited testing since then but appears to be working so far. Maybe that commit has not been merged in the main branch yet? I didn't bother digging further to see why the update didn't work.

@croadfeldt
Copy link

Hopefully google picks up on this. This fixed my missing video clip issues with my teslacam sentry mode and dashcam mode. More SEO following.

Missing video clip teslacam sentry mode and dashcam using teslausb but json and folder present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests