Does Remix have any protection against parsers/bots that extract data from the loader?

[DarkoreXOR]

I'm wondering if there is any protection against accessing loader() results directly via a special http request like:
http://localhost:3000/hello?_data=routes/_test
which freely returns my json data to be used by parsers or bots.

I would like to provide a separate paid API service and complicate such parsing available through the remix app.

[kiliman]

At the moment, Remix doesn't support Middleware (which would run before your loaders), but it is on the Roadmap. #7642

There are a couple of options:

1. You must protect each API route/loader with an access check. Notice how you don't have to check for the API key. The function will throw a JSON response if API key is invalid.

export async function loader({request}: LoaderFunctionArgs) {
  const apiKey = await requireApiKey(request)

  //... do something with key

}
async function requireApiKey(request: Request) {
  const authHeader = request.headers.get('Authorization')
  const apiKey = await checkAuthHeader(authHeader)
  if (!apiKey) {
    throw json({ error: 'Invalid API key'}, { status: 401 })
  }
  return apiKey
}

2. If you're using a custom adapter, like Express, you can put your API check there before passing the request to Remix. This acts like Middleware.

// server.js
// ...
app.all("*", handleRequest);

function handleRequest(req, res, next) {
  if (req.path.startsWith("/api/")) {
    const authHeader = req.headers["authorization"];
    const apiKey = checkAuthHeader(authHeader);
    if (!apiKey) {
      res.status(401).json({ error: "Invalid API Key" });
      return;
    }
  }
  return remixHandler(req, res, next);
}

[gustavopch]

You're showing how API routes can be protected, but I believe the concern is that people may try to scrape data from the GUI routes via data requests instead of going through the API.

[kiliman]

Well, the solution is for those loaders to be protected by auth cookies or similar.

If you're worried about data scraping from your actual users, then the only solution is to remove the <Scripts> component, and Remix will SSR each page. Then the user will have to extract the data directly from the resulting HTML.

[sergiodxa]

You could also detect if the request comes from a bot and don’t render Scripts, that way normal users will get client side JS and bots only SSR without the inline data from loaders

[DarkoreXOR]

Yes, the main problem is that the UI component receives data through a special request to the remix application, and this request can be used by bots/parsers to retrieve data bypassing html.

I'd like to somehow make it harder for bots to get not json data from loader, but for example ready html or encrypted data that will be decrypted inside the component.

[sergiodxa]

You could detect if the request comes from a bot inside your loaders and throw early

export async function loader({ request }: LoaderFunctionArgs) {
  await preventScrappers(request);
  // code
}

And preventScrappers can use the isbot lib to detect if it's a bot and throw an error.

Note that this could happen without a Remix app, if you have a React app consuming an API that API could be scrapped by a bot the same way, and any protection you may add to an API could be added to Remix.

[DarkoreXOR]

Yes, thanks, I know about the API, it will be a separate microservice in docker.

I just need the Remix application to not pass raw data. In loader I will access my API in docker. But I plan to send data from loader in encrypted form and decrypt it in a component on the client (in the browser). The obfuscated JS code intended for the browser can of course be reversed if they really want to, but they will spend time on it anyway, and the encryption method can be changed from time to time.

So far, this is the best solution in my opinion, in addition to the checks for bots.

[gustavopch]

@DarkoreXOR Maybe you can decrypt the data in the client loader that seats between the component and the server loader. Or you can wait for RSC.

[r07r07]

I was also thinking how to protect raw/plain json data from being parsed by bots.
Not sure if this is a feasible approach:

// First, encrypt the data on server side and send encrypted data to clients.
export async function loader() {
  const data = await getSomethingFromDatabase();
  encryptedData = encrypt(data, privateKey);
  return json({payload: encryptedData})
}

// Then, clients will decrypt the encrypted data in their browsers (which means decrypt it in this component).
export default function Page() {
  const { payload } = useLoaderData();
  const privateKey = getPrivateKey(); 
  data = decrypt(payload, privateKey);
  const obj = JSON.parse(data);

  // ...
}

And it's important to obfuscate the JavaScript file (including variables, strings, and function names) to prevent clients from knowing the private key and decryption algorithm in the source code.

Besides, I noticed that some websites are using a similar mechanism.