Storing data in the file format

[synthsin75]

I think I've figured out a good place to store data. Mike (mkelley) was kind enough to explain what 'loadstring' does. I've tested it and, at least for tables, I can convert them to a string and back again.

Now for general tool use that will work well for saving tables as tool preferences, but for project/file specific data, it needs to be stored in the format (as opposed to littering the project file with text files).

I think the best place in the file format to store stuff is in point group names. Granted this needs a dummy layer, named "Do Not Delete", set to hide in editing view and not render, with a single line segment (two points).

Now these two points can be named with your data strings as many times as needed. Granted the user has to actually follow the name of the layer and NOT DELETE it, but this is the best solution I've found. Until MC provides something better, this is my best bet.

Just thought I'd share in case anyone else may find it useful.

[heyvern]

This does sound promising.

I had thought about using other methods similar to this for saving data. I like this one because the data is "hidden" inside the point group names (not the layer name). Plus an empty vector layer won't take so much space. You can easily add new separate data by adding a new point and point group.

I had stayed away from this technique due to the dangers of data loss from accidental deletion of that layer. Layer deletion CAN NOT BE UNDONE in AS. If you delete a layer there is no way to bring it back without reverting to the last saved version (reopening the file without saving changes). The thought of storing important data in such a fragile location made me nervous. But if people know this and are careful it could work... uh... even I am not always careful... . I've lost work just because I deleted a layer by mistake and had to revert the file.

---------

There is one very important thing to watch out for anyone planning to use this. Avoid "strange" non alphanumeric characters. I don't have a list yet but there are certain characters used in lua and other types of data storage or programming that will make an AS file unusable when saved or actually the characters are stripped out completely when saved. I discovered this when saving data in a notes layer. That experiment failed because you can't change notes text with scripting. I started out by pasting in "code" into the notes text and "broke" the file format.

If you have any characters in the names or editable areas of an AS file that "break" AS from reading in the file it just won't open or worse will crash AS. The best bet for using this is to keep the text as simple and straight forward as possible; no ;, :, %, [], (), ", etc. I don't know for sure if all of those characters are "bad", more testing would be needed.

If you use this technique to save data like tables or values, just use space or comma delimiters. AS will wrap text in quotes within the file format. (I don't remember for sure if even a comma is usable in those cases).

I think AS will "strip out" odd characters not allowed in layer, bones or group names when the file is saved. In my experiments I was using a text editor to "force" the code into the name which broke the file format.

-vern

[synthsin75]

Yeah, I'll have to look into those concerns. But there's no need to add any points. The minimum two points can be named as many times as you want without overwriting any of the old names.

Just tried {}[]():;%>,. and it didn't complain at all. Saved, closed, reopened no problem.

[heyvern]

there's no need to add any points. The minimum two points can be named as many times as you want without overwriting any of the old names.

D'oh! Ha ha! Of course. Just keep naming the same dang points... I have problems thinking outside the box sometimes.

Just tried {}[]():;%>,. and it didn't complain at all. Saved, closed, reopened no problem.

Cool! Maybe it was just the notes layer? It figures I would pick the one thing to test this with that can't use funny characters.

-vern

[synthsin75]

Just had some cool ideas. Storing data this way would make it easy for values set by a tool to be used by a layer script. This would make it easier to have 'animated' values for the layer script based on the frame number. You'd just store a table of frame/value pairs as a string point group name with the tool. Then as the layer script runs, it uses this table to adjust that value depending on the frame.

Just brainstorming here, but very exciting idea.

Also, if more scripters than just me intend to use this, we might as well use the same layer. So we need a best practices much like we use for preferences. Start your strings with something like 'syn_tool.value' to distinguish possible multiple uses of 'value'.

This has distracted me, so I've been working up methods(code) for reading and writing to this file location. If this sounds like something others are likely to make use of, I'll post the code snippets for handling this stuff.

[heyvern]

Also, if more scripters than just me intend to use this, we might as well use the same layer. So we need a best practices much like we use for preferences. Start your strings with something like 'syn_tool.value' to distinguish possible multiple uses of 'value'.

This has distracted me, so I've been working up methods(code) for reading and writing to this file location. If this sounds like something others are likely to make use of, I'll post the code snippets for handling this stuff.

I'm on board. Your idea about storing tool info with key frames is really intriguing. I would love to see your code snippets.

I also agree we should agree on a "standard". If the next version of AS has a place to store things it won't be that big a loss. I think it's worth risking accidental data loss to store things in the layer. Hopefully it will be a temporary solution.

-vern

[synthsin75]

Vern,

I'll post some of the code later tonight. I haven't tested it all, and haven't got it all figured out yet. Just been sort of brainstorming most of it.

As far as storing values by frame goes, I'm not sure it'd be useful for tools. What I imagine is user defined settings (set using the tool options of a special tool) that would be used by a layer script. This way some settings, like maybe your gravity script, could be changed easily and perhaps alter over time.

These wouldn't have any connection to any keyframes. The value would just change depending on which frame it's on. Like: value = table[moho.frame]. It's just that each frame index of the table could be set to a custom value by the user.

No idea where you'd use this yet, but it sounds cool as hell anyway.

[synthsin75]

Okay, here's what I have so far. Hopefully I've commented it enough to be clear. This is mainly just to test and show its use.

Code: Select all

-- **************************************************
-- Provide Moho with the name of this script object
-- **************************************************

ScriptName = "Z_ScriptTesting"

-- **************************************************
-- General information about this script
-- **************************************************

Z_ScriptTesting = {}

function Z_ScriptTesting:Name()
	return "Script Testing"
end

function Z_ScriptTesting:Version()
	return "0.0"
end

function Z_ScriptTesting:Description()
	return "Script testbed"
end

function Z_ScriptTesting:Creator()
	return "Synthsin75"
end

function Z_ScriptTesting:UILabel()
	return("Script testing")
end

-- **************************************************
-- Recurring values
-- **************************************************


-- **************************************************
-- The guts of this script
-- **************************************************

function Z_ScriptTesting:IsEnabled(moho)
  return true
end

function Z_ScriptTesting:OnMouseDown(moho, mouseEvent)
  
  local needlayer = true
  
  self:scan(moho)
  
  for i,v in pairs(tbl) do
    if (v:Name() == 'Do Not Delete!!') then
      needlayer = false
      break
    end
  end
  
-----------------create storage layer(if it doesn't exist) and move it to the bottom ---------------------------------
   
  if (needlayer == true) then
    moho:CreateNewLayer(1)
    layer = moho.layer
    layer:SetName('Do Not Delete!!')
    layer:SetRenderOnly(true)
    layer:SetEditOnly(true)
    local mesh = moho:Mesh()
    local Vec1 = LM.Vector2:new_local()
    local Vec2 = LM.Vector2:new_local()
    Vec1:Set(0,0)
    mesh:AddLonePoint(Vec1, 0)
    Vec2:Set(.25,.25)
    mesh:AppendPoint(Vec2, 0)
    mesh:SelectAll()
    moho:PlaceLayerBehindAnother(moho.layer, tbl[table.getn(tbl)])
    
  end
----------------------------table to string-----------------  
  datatbl = {6, 8, 10}
  
  for i,v in pairs(datatbl) do
    if (i == 1) then
      tblstring = v
    end
    if (i > 1) then
      tblstring = tblstring..', '..v
    end
  end
  
  tblstring = 'datatbl = {'..tblstring..'}'
  
  print(tblstring)
  
----------------------------store the string----------------

  self:scan(moho)
  
  for i,v in pairs(tbl) do
    if (v:Name() == 'Do Not Delete!!') then
      moho:SetSelLayer(v)
      mesh = moho:Mesh()
      mesh:SelectAll()
      mesh:AddGroup(tblstring)
    end
  end

----------------------------get string---------------------

  datatbl = nil
  tblstring = nil         --to be sure it's cleared
  
  for i,v in pairs(tbl) do
    if (v:Name() == 'Do Not Delete!!') then
      moho:SetSelLayer(v)
      mesh = moho:Mesh()
      grpid = mesh:CountGroups()-1
      for i = 0, grpid do
        grpname = mesh:Group(i):Name()
        if (string.find(grpname, 'datatbl.+')) then
          tblstring = grpname
        end
      end
    end
  end
  
-----------------------------string to table----------------
  
  stringtbl = loadstring(tblstring)
  stringtbl()
  for i,v in pairs(datatbl) do 
    print(i..'-'..v)
  end
  
  MOHO.Redraw()
  moho:UpdateUI()
end

function Z_ScriptTesting:scan(moho)
  document = moho.document
  layerid = document:CountLayers()-1
  tbl = {}
  for i = layerid, 0, -1 do
    local layer = document:Layer(i)
    table.insert(tbl, layer)
    if(layer:IsGroupType()) then
      local parent = moho:LayerAsGroup(layer)
      self:subscan(moho, parent, tbl)
    end
  end
  return moho, parent, tbl
end

function Z_ScriptTesting:subscan(moho, parent, tbl)
  layerid = parent:CountLayers() -1
  if (layerid > -1) then
    for i = layerid, 0, -1 do
      local layer = parent:Layer(i)
      table.insert(tbl, layer)
      if (layer:IsGroupType()) then
        local parent = moho:LayerAsGroup(layer)
        Z_ScriptTesting:subscan(moho, parent, tbl)
      end
    end
  end
  return moho, parent, tbl
end

[heyvern]

These wouldn't have any connection to any keyframes. The value would just change depending on which frame it's on. Like: value = table[moho.frame]. It's just that each frame index of the table could be set to a custom value by the user.

Ah! Different minds different ideas!

I was thinking that a key on a frame for a bone controlled by a script could control a value in the script but not actually change the value of a bone.For example suppose a "special tool" would use a "script widget" to add values based on the frame to the data storage?

You can add your own custom widgets to the top of a tool "window" (like the boxes for translation or rotate bone tools). These widgets won't do anything. The values however could be stored in the data storage layer to control the function of a layer script.... Wow! THIS is a COOL idea!

With this concept you can KEY FRAME CHANGES TO A LAYER SCRIPT!!!!!

I could never do this with layer scripts because there was NO WAY to store the unique frame data!! With this data storage layer you can now create all kinds of separate values to store changing frame data using a special tool associated with a a layer script!!!!! In the past I have always looked for elements that exist to add data to that is saved in the key frame (values like dynamic bones, or scaling etc).

This could work VERY WELL with my gravity and physics scripts. There could be a widget as part of they "physics tool" that would change it from an "up down" physics simulation to an over head "billard ball" type physics without ever having to touch the script!

You could turn collision on or off or change the gravity or weight of specific bones using values or checkboxes that are part of the "special tool" without needing a bunch of bones to control this!!!!!

Wooohoooo! My brain is spinning.

EDIT: you can easily see the key frames for a special tool by selecting the data storage layer. Key frames could be stored there by keying elements of point/s used for data storage.

Use the "special tool" to slide those keys and it "slides the keys" stored in the data in the data storage layer.

-vern

[synthsin75]

EDIT: you can easily see the key frames for a special tool by selecting the data storage layer. Key frames could be stored there by keying elements of point/s used for data storage.

That's just what I was thinking today! This wouldn't work for tools, but layer scripts could repurpose any of the animation channels on the storage layer. I'm not sure how you'd go about using anything but the boolean channels though.

Might also be able to have the lua console print out a legend that can be aligned to the timeline window to show what each channel in the storage layer does.

I'm already working on a 'layer script enable/disable' tool/layer script combo to test this stuff. I've got more ideas I'll post later tonight.

[heyvern]

Here's a more step by step process for how I think this might work. I have an idea to address using channel keys for more than boolean values. I will use my physics script for the example.

Physics Layer Script Custom Tool
This custom tool will be used only in conjunction with the physics layer script. It will have various entry boxes and check boxes at the top of the window to change only physics properties that are used by the physics layer script. Just like the other standard tools. These "widgets" would not effect any existing animation channels in AS. They would just be a way to set the values used by the physics script.

In the AS script interface you can create these widgets for any tool and they have NO association or connection to anything in AS. You just create an entry box or check box "widget" for the tool. They can contain any value you want and are associated with a variable used by the data storage script and the layer script. These widgets don't have to use any channels associated with standard animation properties in AS except to indicate that a key is on a frame for visual feed back.

So let's say on frame 30 you use the "Physics Tool" to change the gravity of the simulation from "1" to "-1" to make everything "float up". The frame number and value associated with the gravity variable is appended to the "name" of one of the point groups in the data storage layer. The front portion of the names of these point groups could have some type of identifier to link it to a script. As you key frame values for the script, that data is just "added" to the end of the name. If you go back and edit a frame, the script would just look for that frame number to determine if the value currently exists for that frame and change it. If no key exists it adds it between keys that exist or appends it to the end.

A key could be added to any animation channel for the point group points. It just needs a key of any type so AS can "show" that key purely as a visual reference for the user and the script can evaluate the value that is stored in the group name.

When the animation plays, the physics script looks for the gravity value "keys" listed in the data storage layer. When it hits frame 30 the new value is "keyed" and it uses that value for gravity which causes everything to "float" up since the value has changed from "1" to "-1".

Moving keys on the data storage layer
Moving or deleting keys on the data storage layer could be an issue. I have an idea for that... not a great idea but it might not be too bad.

Since I don't think the script interface can "see" when you move keys to send feed back to the layer script or data storage script, I am thinking of using a "pop up" scripted interface window.

You select any frame that has a key on the data storage layer and a pop could be activated with the physics tool. It would allow you to move keys using a button or arrow keys or typing in a value. This pop up interface would interact with both the physics script and the keys in the data storage layer. If you are on a frame with a key you "move" the key using the pop up. This pop up changes the group name to update the new key frame location and at the same time deletes and "moves" the animation channel key that is an indicator of the scripted key information on the data storage vector layer.

This is harder than just dragging the keys but I think it wouldn't be totally "bad".

Data Storage "utility" script
I think this might be a way to go for keeping the layer scripts smaller and "reusing" the functions involved more efficiantly.

All the code needed to store and retrieve data from the "data storage" layer can be in a single utility script in the AS application scripts directory. A layer script will only have to call a single function to store or retrieve data. This utility script is loaded when AS is launched and then the layer scripts won't need all that extra code in them.

The LM tools use this and Fazek used this as well. The tools call often used functions from the utility script. For example the LM function for building a menu of bones or point groups is in the utility script. It's a bunch of code but the tools only have to say "build a menu" and the new menu is returned.

Using this you would only need one line of code calling a function with some arguments in a layer script to save or retrieve data.

-vern

[heyvern]

Ha!

Instead of a pop up to move keys, a key board short cut could be used instead. The custom tool would have a key down for moving key frames. So maybe you press "CTRL + Left arrow" to move a key frame back.

This might be easier to move keys than a pop up but it wouldn't allow moving keys over many frames or "past" other keys easily. Maybe this is added to the "top" of the tool? A "move key" entry box. Type in the new frame number and it moves the key.

I like these ideas.

-vern

[synthsin75]

Using this you would only need one line of code calling a function with some arguments in a layer script to save or retrieve data.

Damn, you're just a half step ahead of me with most of these ideas. By the time I get on here to post them, you already have. I assume that means we're on the right path.

I'm already going to write this to a utility script which will also include my functions for building the layer object table I'm using to find the bottom of the layer stack and cycle through all the layers to find the storage layer. I also thought a utility script would be a good idea since it will standardize our use of this.

Point groups can be used without limit, but repurposing the timeline may not be able to be shared easily. I had originally thought that all of this could be stored on one layer, but we may end up needing a separate layer for each scripter. That may not be so bad since there are so few of us, but that doesn't sound like a long term solution either.

Would we need to go so far as having a separate storage layer for each layer script? For separate scripter layers, you could pass an identifying string to the layer creation function. So 'Syn' could be concatenated like 'Do Not Delete!! -Syn'. This identifier would have to be passed to find the storage layer as well.

If we don't want to try to minimize how many of these layers a project may use, we'd be free to do our own utilities to handle this.

And there's no real appending to these group names. Only delete and make a new one that includes the appending data.

Let me know what you think.

[heyvern]

Would we need to go so far as having a separate storage layer for each layer script?

I don't think so. All you need is an identifying label for point groups. At mostt I think each scripter could have their own set of points not layers. I still think this can be done with one vector layer. Remember that whatever text we use in the point groups name can include the scripter's identification as well as any other information.

As long as the groups are specifically labeled they could be "invisible" to each other.

-vern

[synthsin75]

I was thinking of multiple layers for repurposing the timeline only. I just can't think of anyway to keep the keyframe indicators separate. The only reason I started thinking about repurposing the storage layer's timeline is because I'm not real sure how a tool's options could be updated while scrubbing the timeline.

Say you have a check box that is alternately checked and unchecked as the timeline plays. The data that the layer script is reading is changing, but could the tool reflect these changes. Even if the tool reads in the same data for the check box status, can tools update when they're not being used?

This is the only reason I'd really use keyframe indicators.

So maybe have one data storage layer, but have a control/indicator layer for each layer script that needs one? These could be created by the associated tool, I guess.

But yeah, no reason that simple data storage can't be shared on one layer.