Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apscheduler external action #174

Open
precifiqueja opened this issue Jan 8, 2021 · 0 comments
Open

apscheduler external action #174

precifiqueja opened this issue Jan 8, 2021 · 0 comments

Comments

@precifiqueja
Copy link

precifiqueja commented Jan 8, 2021

Resultados de tradução
Hello guys, good morning, good afternoon, good night.

I would like to know if there is any possibility of creating help me to fit a part in the code. As I create many tarafas, and they change constantly, from spider parameters etc. I created a simple code that inserts in the database more quickly. However, only the tasks were created, and the apscheduler is fantastic, and I don't know how to connect them automatically without having to edit the task.

So after I create the tarafera externally, it has a stopped status, and I already tried to use this endpoint (http://127.0.0.1:5000/1/tasks/xhr/fire/1*) without success.

Below my code used to enter the database

from pypika import Query, Table
import sqlite3
import os
from datetime import datetime


class ScrapywebDB:
    def __init__(self, spider=None, project=None):
        self.spider = spider
        self.project = project
        self.name_string = None
        self.settings_arguments = None
        self.user_id_list = []
        self.conn = sqlite3.connect(os.path.abspath('timer_tasks.db'))
        self.AddNewSpiderCli()

    def GetUsers(self):
        user_id_list = ['2',
                        '4',
                        '10',
                        '11',
                        '13',
                        '14',
                        '15',
                        '17',
                        '18',
                        '19',
                        '20',
                        '21',
                        '23',
                        '33',
                        '37',
                        '39',
                        '40',
                        '43',
                        '44',
                        '45',
                        '46',
                        '47'
                        ]
        self.user_id_list = user_id_list
        return user_id_list


    def AddNewSpiderCli(self):
        self.GetUsers()
        task = Table('task')
        spider = self.spider
        project = self.project
        agora = datetime.now()
        create_time = str(agora.strftime("%Y-%m-%d %H:%M:%S"))
        jobid = str(agora.strftime("%Y-%m-%dT%H_%M_%S"))
    
        def QueryBuilder():
            q = Query.into(task).columns(
                'name', 'trigger', 'create_time', 'update_time', 'project',
                'version', 'spider', 'jobid', 'settings_arguments', 'selected_nodes',
                'year', 'month', 'day', 'week', 'day_of_week', 'hour', 'minute', 'second',
                'timezone', 'jitter', 'misfire_grace_time', 'coalesce',
                'max_instances').insert(
                name_string, 'cron', create_time, create_time, project,
                'default: the latest version', spider, jobid, settings_arguments,
                '[1]', '*', '*', '*', '*', '*', '*', '15,30,45,59',
                '0', 'America/Sao_Paulo', 0, 600, 1, 1)
    
            q = str(q).replace('\"', '`').replace("\'", "\"")
            return q
    
        for item in self.user_id_list:
            user_id = item
            name_string = 'user_id_%s_%s' % (user_id, self.spider)
            settings_arguments = str({
                "SEARCH": "all",
                "URL_SEARCH": "americanas",
                "USER_ID": user_id, "page_range": "[]",
                "page_total": "None",
                "node_ip": "127.0.0.1:6800",
                "setting": ["CONCURRENT_REQUESTS=1"]
            })
    
            cur = self.conn.cursor()
            query = QueryBuilder()
            print(query)
            cur.execute(query)
            self.conn.commit()


ScrapywebDB(spider='PDU', project='b2w')

image

I hope you can help me. Tanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant