Is there a documented way to scrape Single Page Applications? #114

Activity

1. bilogic

   added

   documentationImprovements or additions to documentationImprovements or additions to documentation

   on Apr 8, 2023
2. bilogic

   assigned

   ksassnowski

   on Apr 8, 2023
3. ksassnowski commented on Apr 11, 2023

   

   ksassnowski

   on Apr 11, 2023

   Contributor

   More actions

   If the CSRF token is part of the page's source, then you can extract it like any other piece of information. You would then have to figure out how exactly the site expects the CSRF to be sent with each subsequent request, for example as a header.

   You can then set the header from within your spider before dispatching new requests: https://roach-php.dev/docs/processing-responses#returning-custom-requests

   So, assuming the CSRF token exists in the page source like this

   <meta name="csrfToken" content="...">

   Your parse method could look something like this

   public function parse(Response $response): \Generator
   {
       // do your scraping here...
   
       $csrfToken = $response->filter('meta[name="csrfToken"]')->attr('content');
   
       $request = new Request(
           'POST', 
           'https://next-url-to-crawl.com',
           $this->parse(...),
           // Assuming the csrf token should get passed in the X-CSRF-Token header
           ['headers' => ['X-CSRF-Token' => $csrfToken]],
       );
   
       yield ParseResult::fromValue($request);
   }
4. roach-php

   locked and limited conversation to collaborators on Jun 3, 2023
5. ksassnowski

   converted this issue into a discussion #131 on Jun 3, 2023