Diligent Dilettante

ZettelKasten part 5

An API worth making requests to

This is a followup to a previous article, ZettelKasten Part 4

In the previous article I went down a huge rabbithole but at least I updated my Grakn version. In this part we'll be updating previous API methods to work with this version of Grakn and then getting athe API more complete.

Updating the API

So now that I've got my grakn instance running a higher instance type, time to downgrade it. Because the node client isn't currently at the bleeding edge. But we'll at least get as close to bleeding as possible without requiring a tournequette.

docker run --name grakn -d -v $(pwd)/db:/grakn-core-all-linux/server/db/ -p 1729:1729 graknlabs/grakn:2.0.0-alpha-3

Then in my directory for the zettelkasten server:

yarn upgrade grakn-client@2.0.0-alpha-7

This is to save you a bunch of pain if you're actually following along and don't want arbitrary functions to not work randomly due to a mismatch in the RPC protocol between the server and the client.

Updating the Schema

Now we have to rewrite our schema to make a few particular changes, most of which are visible in this set of code:

zettel sub entity,
  owns content,
  owns date,
  owns name,
  plays zettel-citation:cites,
  plays zettel-link:linked,
  plays zettel-link:linker,
  plays zettel-tagged:tagged,
  owns uuid @key;
  1. owns is the new has. has is now for Things (like instances) and owns is for Types (like classes).
  2. Relations now have scoped roles. Which means that we can stop giving terrible names to all our relations but instead we have to prefix them with the relation they're scoped to. I knew this was coming so I named them smartly before now.
  3. @key replaces key, so now we properly say owns uuid instead of key uuid and @key is a little decorator at the end there.

The other change can be observed here:

uri sub attribute, abstract,
  value string;

You won't find that one in their documentation. You can now only subtype attributes which are abstract. This explicitly contradicts Grakn's documentation and it's probably a bug.

Update the migration script

We had our little script that did the migration, now's an excellent time to visit it and see how the differences in the new version of Grakn affect it.

First the imports:

import Path                 from 'path';
import {Grakn, GraknClient} from 'grakn-client/dependencies_internal';
import {readFileSync}       from 'fs';

Yeah, it's a little wonky, reaching right into that dependencies_internal file. Hopefully a better way comes along.

  const database = 'zettelkasten';
  const graql   = readFileSync(Path.join(__dirname,
                                         './migrations/zettelkasten1.graql'), 
                                         'utf8')
  const client  = new GraknClient('localhost:1729');
  const databaseExists: boolean = await client.databases().contains(database);
  if (!databaseExists) {
    await client.databases().create(database);
  }
  const session = await client.session(database, Grakn.SessionType.SCHEMA);

  const writeTransaction = await session.transaction(
    Grakn.TransactionType.WRITE
  );
  await writeTransaction.query().define(graql);
  await writeTransaction.commit();
  await writeTransaction.close()
  await session.close();
  client.close();

I was going to just show the lines that changed, but that's basically all of them.

The port has changed to 1729.

There's a DatabaseManager which I get with client.databases() in order to create the database if I don't find it.

The Session now requires a Grakn.SessionType.

The Transaction now requires a Grakn.TransactionType instead of being session.transaction().write().

A QueryManager is now summoned with transaction.query() and is used to issue particular types of queries, in our case define() with the text of our graql query.

I now await for the Session to close (I maybe should have been already). I also close the Transaction but I'm not sure I need to after the commit. You no longer need to await for the client to close - it's synchronous.

Update the API itself

The API needed extensive updates but I also made some parts nicer.

First, the imports:

import Koa, {Context as KoaContext, Middleware, Next} from 'koa';
import {GraknClient} from 'grakn-client/dependencies_internal';
import {Grakn}       from 'grakn-client/Grakn';

I've renamed Context because I'm going to have my own Context to make my typescript stuff easier to work with.

const graknClient = new GraknClient('localhost:1729');
const database    = 'zettelkasten';

interface Context extends KoaContext {
  graknSession?: Grakn.Session,
  graknTransaction?: Grakn.Transaction
}

Our session middleware is mostly unchanged:

const graknSessionMiddleware: Middleware = async (ctx: Context, 
                                                  next: Next) => {
  ctx.graknSession = await graknClient.session(database, 
                                               Grakn.SessionType.DATA);
  await next();
  await ctx.graknSession.close();
  delete ctx.graknSession;
};

And our Write and Read middleware are also mostly unchanged. Mostly just changes to use the TransactionType now.

const writeTransactionMiddleware: Middleware = async (ctx: Context, 
                                                      next: Next) => {
  ctx.graknTransaction = await ctx.graknSession.transaction(
    Grakn.TransactionType.WRITE);
  await next();
  await ctx.graknTransaction.commit();
  delete ctx.graknTransaction;
}

const readTransactionMiddleware: Middleware = async (ctx: Context, 
                                                     next: Next) => {
  ctx.graknTransaction = await ctx.graknSession.transaction(
    Grakn.TransactionType.READ);
  await next();
  await ctx.graknTransaction.close();
}

Let's take a look at the new uses of the Concepts API. In particular, the POST for making a new zettel

    const zettelRequest = ctx.request.body;
    const transaction = ctx.graknTransaction;
    const concepts = transaction.concepts();
    const ZettelType = await concepts.getEntityType('zettel');
    const ContentType = await concepts.getAttributeType('content');
    const DateType = await concepts.getAttributeType('date');
    const NameType = await concepts.getAttributeType('name');
    const UUIDType = await concepts.getAttributeType('uuid');
    const content = await ContentType.asRemote(transaction).asString()
      .put(zettelRequest.content);
    const date = await DateType.asRemote(transaction).asDateTime()
      .put(new Date());
    const name = await NameType.asRemote(transaction).asString()
      .put(zettelRequest.name);
    const uuid = await UUIDType.asRemote(transaction).asString()
      .put(UUID());
    const zettel = await ZettelType.asRemote(transaction).create();
    const remoteZettel = zettel.asRemote(transaction);

Can you see why I want to write a more fluent API? So many awaits. But there's more to come. For now, let's look at how this works now.

We've got this Manager pattern again with ConceptManager which is what we get from transaction.concepts(). We then get Types for zettel, content, date, name, and uuid. However, now we specify the exact Type because Grakn now enforces this behavior. We then convert our local concepts into remote concepts with asRemote(transaction) - note that this just means we have access to up-to-date information on the concept and some more interesting features like creating some. Each of these attributes then has as with some type called on it, like asString() - this way we get a StringAttributeType for example - just more type enforcement. With this we call put(value) with the values we want. Note this doesn't actually take effect until we commit() the transaction, which occurs at the end of the write middleware.

The put calls give us a Thing, particularly an Attribute corresponding to the AttributeType we were interacting with. Again, the distinction to be made: A Type is schema information, the shape of data, a class in essence. The Thing is data, an instance essentially.

Once we've got that, we make our Zettel using our ZettelType which is an EntityType. EntityTypes use create() instead of put because Entitys don't contain values, they just has Attributes they have, which conform to the the AttributeTypes described as owns on the EntityType.

And that hasing is what we do next.

await Promise.all([
  remoteZettel.setHas(content), remoteZettel.setHas(date),
  remoteZettel.setHas(name), remoteZettel.setHas(uuid)
]);

And then one of the reasons I want to make a nicer serializer:

    const contentObject = {type: 'attribute', label: 'content', 
      id: content.getIID(), value: content.getValue()};
    const dateObject = {type: 'attribute', label: 'date', id: date.getIID(), 
      value: date.getValue()};
    const uuidObject = {type: 'attribute', label: 'uuid', id: uuid.getIID(), 
      value: uuid.getValue()};
    const nameObject = {type: 'attribute', label: 'name', id: name.getIID(), 
      value: name.getValue()};
    const zettelObject = {type: 'entity', label: 'zettel', 
      id: zettel.getIID(), attributes: {
        uuid: uuidObject, name: nameObject, 
        date: dateObject, content: contentObject}
    };

Now for the thing that reads all the zettel.

    const transaction = ctx.graknTransaction;
    const ZettelType = await transaction.concepts().getEntityType('zettel');
    const zettelInstancesStream = ZettelType.asRemote(transaction)
      .getInstances();

So far so good.

    //this won't be necessary once Grakn core alpha-9 has a node client
    //implementation
    //TODO get rid of this mess once on core alpha-9
    const zettel     = await Promise.all(await ZettelType
      .asRemote(ctx.graknTransaction).getInstances()
        .map(eachZettel => eachZettel.asRemote(transaction).getHas()
          .collect().then(attributes => Promise.all(attributes.map(
            attribute => attribute.asRemote(transaction).getType().then(
              attributeType => ([
                attributeType.getLabel(), 
                attribute.getValue()
              ])
            )))
            .then(entries => entries.reduce((object, entry) => {
              object[entry[0].toString()] = entry[1];
              return object;
            }, {type: 'zettel'}))
         )).collect());

WTF is going on here?

Well, a lot. And this right here is why I want to write a more fluent Grakn Node.js Client API. To summarize: I want to get a nice set of zettels with all their attributes. To do that, I need to

  1. Get all the instances of zettels
  2. For every Zettel get all its attributes
  3. For every attribute get its label and value
  4. Make an object out of the attributes
  5. Return an array of all that stuff.

And you can see also I serialized it entirely differently here. I'm going to write a standard serialier for entities that'll take care of this by standardizing how we do this.

In the future, as noted by the big comment, we won't need to do this because we'll get the type with every attribute we get, so we don't need to turn around and ask for it.

Improving the API

Serialization and Deserialization

Let's write a serialization system. I want it to apply generally to any entity in Grakn. We're not going to worry about relations for now, it just needs to make a nice object out of an entity with the attributes.

I imagine that a Zettel should look like this:

{
  "uuid": uuid,
  "name": name,
  "content": content,
  "date": date
}

Something like that. I'm not going to bother putting in the fact that its a zettel - that's apparent from the API resource location. So I'll avoid metadata. Basically just stick the attributes on there, {label: value}.

However, an interesting concern arises. Our schema says content is not just a StringAttribute - it also has metadata about the content: mime-type. So we'll need the ability for attributes to have a value and their own attributes. So we'll do it this way:

{ 
  "uuid": {
    "value": uuid
  },
  "name": {
    "value": name
  },
  "uuid": {
    "value": uuid
    "mime-type": {
      "value": mimeType,
      "mime-type-type": {
        "value": mimeTypeType
      },
      "mime-type-subtype": {
        "value": mimeTypSubtype
      }
    }
  },
  "name": {
    "value": name
  }
}

Hmmm, ugly. Not great. We maybe should go for something more universal.

{
  type: {label},
  has: [
    {
      type: {label},
      value
    }
  ],
  relations: [
    {
      type: {label},
      playersByRoleType: {
        [roleType] => Thing
      }
    }
  ]
}

This is at least a more universal way of serializing we can use all sorts of places. Important notes:

  1. We'll need to detect when recursion occurs and avoid it.
  2. We need to determine how far to go when reading records to avoid just traveling down some long path.
  3. We can probably use this to also deserialize records.

This looks awfully close to an interface in Typescript, so let's make interfaces for our main friends. Entity, Attribute, and Relation, and Thing - probably this is the start to a more fluid API. We'll just build it as we need it and split it out when it makes sense.

So we have a SerializedType

export interface SerializedType {
  label: string
}

And SerializedThings

import {SerializedType} from './SerializedTypes';

export interface SerializedThing {
  iid?: string,
  type: SerializedType, // for now we don't need to distinguish Thing from any
                        // other Type
  has?: SerializedAttribute[],
  relations?: SerializedRelation[]
}

export interface SerializedAttribute extends SerializedThing {
  value: string | Date  | number | boolean
}

export interface SerializedRelation extends SerializedThing {
  playersByRoleType: {
    [roleType: string]: SerializedThing
  }
}

Now we just need something that will serialize Things.

import {SerializedAttribute, SerializedThing}    from './SerializedThings';
import {SerializedType}     from './SerializedTypes';
import {Thing, Grakn, Type, Attribute, AttributeType} from 'grakn-client/dependencies_internal';

export const serializeThing = async (thing: Thing, transaction: Grakn.Transaction
    ): Promise<SerializedThing> => {
  const remoteThing = thing.asRemote(transaction);
  //this won't be necessary once Grakn core alpha-9 has a node client
  //implementation
  const type = await remoteThing.getType().then(serializeType);
  const has: SerializedAttribute[] = await Promise.all(
    await remoteThing.getHas().map(
      attribute => serializeAttribute(attribute, transaction)
    ).collect()
  );
  let obj: SerializedThing = {type};
  if (has.length > 0) {
    obj = {...obj, has};
  }
  return obj;
}

export const serializeAttribute = async (attribute: Attribute<AttributeType.ValueClass>,
    transaction: Grakn.Transaction): Promise<SerializedAttribute> => {
  const thing = await serializeThing(attribute, transaction);
  const value = attribute.getValue();
  return {...thing, value};
}

export const serializeType = (type: Type): SerializedType => {
  return {label: type.getLabel()};
}

There we go, and now to update our GET /:

  .get('/', async (ctx: Context) => {
    const transaction = ctx.graknTransaction;
    const ZettelType = await transaction.concepts().getEntityType('zettel');
    const zettel = await ZettelType.asRemote(ctx.graknTransaction)
      .getInstances().map(
        eachZettel => serializeThing(eachZettel, transaction)).collect();
    ctx.body = zettel;
  })

And our POST /:

    await Promise.all([
      remoteZettel.setHas(content), remoteZettel.setHas(date),
      remoteZettel.setHas(name), remoteZettel.setHas(uuid)
    ]);
    const serializedZettel = await serializeThing(remoteZettel, transaction);
    ctx.body = serializedZettel;

Alright, we're on a roll.

I'm going to write another post tonight or tomorrow as I add:

  1. Updates
  2. Delete
  3. Show individual Zettel

The adventure continues in ZettelKasten Part 6