Challenge Bot - Request for Proposals (RFP)

[rdearman]

I've been having some discussion of a new more "generic" challenge bot since Twitter is pulling the free access.

So I was simply going to port the SC bot over to Mastodon API. But smallwhite's suggestion of combining challenge bots into a generically maintained bot, and also data input points not reliant on a commercial or single source. So I'm starting this thread to gather ideas about what this type of bot might look like. Feel free to join the discussion, both technical and non-technical views are welcome.

My thinking is the "bot" would have the following features/design.

A database (MySQL or PostgreSQL) which would have the following things:

• A Challenge Metadata set of tables, which would allow the input of things like,
  -- input locations: email, webpage, Mastodon keys, WhatsApp phone number, etc.
  -- A definition of the challenge scoring system.
  -- A definition of the challenge display system (webpage, embedded chart).
  -- Challenge dates, e.g. start and end dates.
  -- Challenge controls, crontab type data, how frequently to poll data sources, etc.
  
• User data area, name, nickname, email address, WhatsApp phone number, Mastodon nickname, etc.
• Lookups area. Language names and ISO codes, Language families and ISO codes,
• Data area which will contain the actions users have taken, read X books, wrote Y words, studied Z hours, etc.

After we get a fair definition I'm happy to open a github repository and add it into the language group (where we keep our version of PhpBB, the current SuperChallenge bot, etc)

I don't particularly like the current SC Bot use of stored procedures in MySQL, and I would prefer PostgreSQL, but that is just a personal preference.

It might be useful to create a REST API for the generic bot, so people could develop Android or iPhone apps if they were that way inclined.

**I am aware this isn't really true RFP, but it sounded nice

[rdearman]

I'm going to suggest using Python for this (even though I think whitespace delimited programming languages are an abomination) since there are a lot of pre-built libraries for email and web access. It supports IMAP email protocol.

Code: Select all

import imaplib
import email

def extract_body(payload):
    if isinstance(payload,str):
        return payload
    else:
        return '\n'.join([extract_body(part.get_payload()) for part in payload])

conn = imaplib.IMAP4_SSL("imap.gmail.com", 993)
conn.login("user", "password")
conn.select()
typ, data = conn.search(None, 'UNSEEN')
try:
    for num in data[0].split():
        typ, msg_data = conn.fetch(num, '(RFC822)')
        for response_part in msg_data:
            if isinstance(response_part, tuple):
                msg = email.message_from_string(response_part[1])
                subject=msg['subject']                   
                print(subject)
                payload=msg.get_payload()
                body=extract_body(payload)
                print(body)
        typ, response = conn.store(num, '+FLAGS', r'(\Seen)')
finally:
    try:
        conn.close()
    except:
        pass
    conn.logout()

[CDR]

Warning: I've mostly programmed scripts for myself, I've only ever had some internal tools I created bounced around my company by word-of-mouth.

+1 for the idea and your solution!
+1 for Python, for the same reasons you stated & it's the language I have the most experience in
+1 for REST API

If my skillset can manage it, I would be happy to program support for different ways of submitting. I would say Google Form/Spreadsheet sounds the least daunting for me to start.

If I cannot manage that, at the least I would be happy to test and try to find bugs.

[rdearman]

Warning: I've mostly programmed scripts for myself, I've only ever had some internal tools I created bounced around my company by word-of-mouth.

+1 for the idea and your solution!
+1 for Python, for the same reasons you stated & it's the language I have the most experience in
+1 for REST API

If my skillset can manage it, I would be happy to program support for different ways of submitting. I would say Google Form/Spreadsheet sounds the least daunting for me to start.

If I cannot manage that, at the least I would be happy to test and try to find bugs.

Cool. I have partially completed an email bot for the Super challenge which I hope to finish this evening (in python). Because it is a running challenge and I only have until the 9th of February I plan to hack that together regardless. I will do some basic testing tomorrow and then I will publish the email address.

The only issue is that we never took email address for the SC bot but I will need it to verify people. I plan to add a sign-up keyword to let people reapply with there email

[smallwhite]

• Data area which will contain the actions users have taken, read X books, wrote Y words, studied Z hours, etc.

I'd like to know about tweet parameters. Does each challenge get to choose its own parameter names and parameter type (eg. alphanumeric, integer, must-be-from-lookup-list...)?

[rdearman]

• Data area which will contain the actions users have taken, read X books, wrote Y words, studied Z hours, etc.

I'd like to know about tweet parameters. Does each challenge get to choose its own parameter names and parameter type (eg. alphanumeric, integer, must-be-from-lookup-list...)?

That would be the idea behind having a challenge profile where you define what input is acceptable. Can you please get an example so I can be sure we are talking about the same thing?

[smallwhite]

• Data area which will contain the actions users have taken, read X books, wrote Y words, studied Z hours, etc.

I'd like to know about tweet parameters. Does each challenge get to choose its own parameter names and parameter type (eg. alphanumeric, integer, must-be-from-lookup-list...)?

That would be the idea behind having a challenge profile where you define what input is acceptable. Can you please get an example so I can be sure we are talking about the same thing?

6WCers report progress by tweeting something along the lines of:
@6WCBot I studied 45 minutes of #Deutsch today.

1.
The 6WC admin will want to define:
MINUTES ... numeric, integer
LANGUAGE ... from ISO lookup table
as a minimum.
(And not be stuck with NUMBER_OF_BOOKS, for example).

2.
On top of that, currently, 6WCers can tag the 45 minutes with participant-defined labels:
@6WCBot I studied 45 minutes of #Deutsch using #Anki today.
where labels #Anki or #Quizlet or even #AnkiNewCards #AnkiMatureCards are up to each participant to name (and are not decided by admin). Each participant can define as many such labels as she wants.

Will admin get freedom 1?
Can participants get freedom 2?

[rdearman]

My current thinking around point one is that when you setup a challenge you define the TOKENS which the bot must recognise in order to process the input. So you define a standard set of regular expressions which the bot will need to process for challenge XYZ. It then parses the string and takes the actions defined for each token. This will be very difficult to make generic, so will require some hard thinking, but it would mean you can use any expression you want. So you would get freedom #1

Your second point about adding tags to each input is again something that would be defined in the meta-data for the challenge. You would have to tell the bot what to do with these keywords. If it is just "add new keywords to the participant keyword table" then that is what would happen. So this isn't a deal-breaker, although I currently don't have any thoughts about how it could be implemented.

The issue here is that I can design a bot to do the SC, I can design a bot to do the Output Challenge or the 365 challenge or the 6Week challenge. But designing a bot that can handle all of them... well that is a lot more difficult.

At the moment I'm just hacking together something that will accept emails for the superchallenge, and that isn't straightforward. After all, surrealix worked on the original bot for almost a year! I've got 4 days and counting to come up with an email replacement. A generic bot will probably take a year. In the meanwhile, you might want to get someone to rework the 6-week challenge bot to accept email.

[smallwhite]

So you are letting us report scores with an English sentence like the 6WC tweets? I think that's too nice of you if it's difficult. I think a barebones CSV kind of format would be good enough, especially if it's just submitted via email and the other participants are not reading it (unlike tweets in 6WC).

How about hosting an input page on language-learners.org?

Challenge Bot - barebones input htm.jpg

[smallwhite]

I think these are all the 6WC fields.

Maybe keeping all past submissions on the page after inputting username / after logging in / after passwordless logging in? We are no longer restricted by the Twitter interface.

Challenge Bot - input htm 6.jpg